Assignment 1 Description
Weighting and Due Date
• Marks for this assignment contribute 5% of the overall course mark.
• Marks for functionality will be awarded automatically by the web submission
system.
• Due dates: Milestone – 11:55pm Tuesday of week 7, Final – 11:55pm Friday of
week 7.
• Late penalties: For each part, the maximum mark awarded will be reduced by
25% per day / part day late. If your mark is greater than the maximum, it will be
reduced to the maximum.
• Core Body of Knowledge (CBOK) Areas: abstraction, design, hardware and
software, data and information, and programming.
Project Description
In this assignment you will implement a tokeniser that with minor changes could be used to complete variations of projects 6, 7, 8, 10 and 11 in the nand2tetris course. A detailed description of the requirements are shown below. The exectuable programs, tokens and tokens-context will read text from standard input and produce a list of tokens in the text on standard output.
Note: you should complete workshop 05 before attempting this assignment.
SVN Repository
You must create a directory in your svn repository
named:
• Makefile – this file is used by make to compile your tokeniser program – do not
modify this file.
• Makefile-extras – this file is used by make to compile your tokeniser program
– do not modify this file.
• tokens – executable script that will run your compiled tokens program – do not
modify this file.
• tokens.cpp C++ source file containing the main() function for tokens – do not
modify this file.
• tokens-context – executable script that will run your compiled tokens- context program – do not modify this file.
• tokens-context.cpp C++ source file containing the main() function for tokens- context – do not modify this file.
• tokeniser.cpp C++ source files containing the next_token() function.
• bin – this directory contains precompiled programs and scripts – do not modify
this directory.
• lib – this directory contains precompiled components – do not modify this
directory.
• includes – this directory contains .h files for the library – do not modify
this directory.
• tests – this directory contains sample test data, it can be used to store any extra
files you need for testing
Note: you must use the following command to add your startup files to your svn
repository:
% svn add Makefile Makefile-extras bin includes lib tests tokens tokens.cpp t
okens-context tokens-context.cpp tokeniser.cpp lib/*/lib.a
Submission and Marking Scheme
Submissions for this assignment must be made to the web submission systemLinks to an external site. assignment named: Assignment 1 – Submit Here. The assessment is based on “Assessment of Programming Assignments”.
Note: the Submit Here assignment will show a breakdown of your marks by category
but it will always show your total mark as 0.
Your tokeniser program must be written in C++. Your tokens and tokens-
context programs will be compiled using the Makefile and
the tokens.cpp or tokens-context.cpp files included in the zip file attached below together with the tokeniser.cpp file in your svn directory. Your programs will then be tested using the set of test files that are attached below. In addition a number
of secret tests will also be run. Note: if your program fails any of these secret tests
you will not receive any feedback about these secret tests, even if you ask! Assignment 1 – Milestone Submissions: due 11:55pm
Tuesday of week 7
The marks awarded by the web submission systemLinks to an external site. for the milestone submission contribute up to 20% of your marks for assignment 1. The marks for the Milestone Tests are used as the marks for the milestone submission. The Milestone Tests test the milestone token definitions shown below using
your tokens program. Your milestone submission mark, after the application of late penalties, will be posted to the myuni gradebook when the assignment marking is
complete.
You can view the Milestone Tests marks in the Milestone assignment but submissions must be made using the Assignment 1 – Submit Here assignment.
Assignment 1 – Final Submissions: due 11:55pm Friday of
week 7
The marks awarded for the final submission contribute up to 80% of your marks for assignment 1.
Your final submission mark will be the geometric mean of three components, the marks for the Final Tests, a mark for your logbook and a mark for your code. It will be limited to 20% more than the marks for the Final Tests. See “Assessment – Mark Calculations” for examples of how the marks are combined. The Final Tests are all
tests run, including those used for the Milestone Tests. Your final submission mark, after the application of late penalties, will be posted to the myuni gradebook when the assignment marking is complete.
You can view the Final Tests marks in the Final assignment but submissions must be made using the Assignment 1 – Submit Here assignment.
Logbook Marking
Important: the logbook must have entries for all work in this assignment, including your milestone submissions. See “Assessment – Logbook Review” for details of how
your logbook will be assessed.
Code Review Marking
For each of your programming assignments you are expected to submit well written code. See “Assessment – Code Review” for details of how your code will be
assessed.
Assignment 1 – Participation Marks
Any submissions to assignment 1 that are made more than one week before the due date for Final Submissions may be awarded up to 10 participation marks. The participation marks will be the marks awarded for the Final Tests divided by 10. You can view the participation marks awarded in the One Week Early assignment but submissions must be made using the Assignment 1 – Submit Here assignment. The participation marks will be allocated to week 6.
Tokenisers
Background
The primary task of any language translator is to work out how the structure and meaning of an input in a given language so that an appropriate translation can be output in another language. If you think of this in terms of a natural language such as
English. When you attempt to read a sentence you do not spend your time worrying about what characters there are, how much space is between the letters or where lines are broken. What you do is consider the words and attempt to derive structure and meaning from their order and arrangement into English language sentences, paragraphs, sections, chapters etc. In the same way, when we attempt to write translators from assembly language, virtual machine language or a programming language into another form, we attempt to focus on things like keywords, identifiers, operators and logical structures rather than individual characters.
The role of a tokeniser is to take the input text and break it up into tokens (words in
natural language) so that the assembler or compiler using it only needs to concern itself with higher level structure and meaning. This division of labor is reflected in most programming language definitions in that they usually have a separate syntax definition for tokens and another for structures formed from the tokens.
The focus of this assignment is writing a tokeniser to recognise tokens that conform to a specific set of rules. The set of tokens may or may not correspond to a particular language because a tokeniser is a fairly generic tool. After completing this assignment we will assume that you know how to write a tokeniser and we will provide you a working tokeniser to use in each of the remaining programming assignments. This will permit you to take the later assignments much further than
would be otherwise possible in the limited time available.
Writing Your Program
You are required to complete the implementation of the C++
file tokeniser.cpp which is used to compile the programs tokens and tokens-context.
You will complete the implementation of a function, next_token(), that will read text character by character using the static function nextch(), and return the next recognised token in the input. The tokens that must be recognised in the milestone and final submissions are specified below and in the file includes/tokeniser.h.
Your tokens and tokens-context programs will be compiled using the Makefile in the zip file attached below using the command:
% make
Note: Do not modify the Makefile, tokens.cpp, tokens-context.cpp or the sub-
directories bin, includes and lib. They will be replaced during testing by the web
submission system.
Testing Your Program
For each file in the tests directory, the output of the tokens and tokens-
context programs must match the corresponding .tokens and .context output files respectively. You must not produce any output of your own. You can both compile and test your programs against all of the supplied tests using the command:
% make
The testing will not show you any program output, just whether or not a test was
passed or failed. If you want to see the actual output, the commands used to run the
tests are shown in string quotes (“). Simply copy the commands between the string
quotes (“) and paste them into your shell.
The web submission system will test your program in exactly this way. The key difference between your testing and the web submission testing is that the web submission system has some secret tests that it will use.
If you want to try additional tests, just create some new files in the tests sub- directory and generate the correct outputs using the command:
% make test-add
This will increase the number of tests that will be run in the future.
Milestone Tokens
Your milestone submission will only be awarded marks for tests that require the correct recognition of the milestone tokens described in
the includes/tokeniser.h file.
Note: the includes/tokeniser.h file describes
• the grammar rules for all tokens,
• the tokeniser interface functions that must be implemented,
• the rules for preprocessing special characters (not required for the milestone),
• the rules for differentiating identifiers and keywords (not required for the
milestone),
• the rules governing the context of some token spelling’s (not required for the
milestone) and
• the rules governing error handling.
Notes: all input must be read using the function next_ch() which must use the external
function read_char().
Final Submission Tokens
Your final submission will be awarded marks for tests that require the correct recognition of all tokens and tokeniser interface functions described in
the includes/tokeniser.h file. Tests
In addition to the test files in the zip file attached below, we will use a number
of secret tests that may contain illegal characters or character combinations that may defeat your tokenisers. The secret tests may also check whether or not you have followed the rules for keyword recognition. Note: these tests are secret, if your programs fail any of these secret tests you will not receive any feedback about these secret tests, even if you ask!
Startup Files
The startup files in the attached zip file should work on most 64-bit Linux systems and on a Mac. Please see the Startup Files for Workshops and Assignments page for more information.
• assignment-tokeniser.zip