Re: A bit confused about the relationship between Tokenizer and Compilation
N.B. I don't participate in the Coursera course in any way. I'm not aware if there are any additional requirements that it has.
The overall goal is to create a compiler, which takes a bunch of JACK files and produces the corresponding .vm files. This is a large and complex task, so it's split in several smaller ones. And you want to be able to test them. That's where these files come to play: the xxxT.xml files are used to test just the Tokenizer. Then the xxx.xml files are used to test the Parser (it should use the Tokenizer directly, not by reading the xxxT.xml files).
In the next assignments you'll start to produce real VM code, again directly, not by reading the xxx.xml files. But it's a good idea to write your tools in such a way that when they are run with some options they will still produce the XML files. This will help you debug problems in the future.
The xxxT.xml files are strictly for debugging the tokenizer. As soon as you are happy with your tokenizer you no longer need to write them. Coursera does not require you to generate them.
Note that when you convert JackAnalyzer into JackCompiler, it may be useful to (optionally) write the xxx.xml files in parallel with the xxx.vm files. Looking at the end of the XML file can help debugging when your compiler crashes deep into recursive parsing.
I wrote an XmlWriter class that takes a file name and used it for both the xxxT.xml and xxx.xml files. By passing in null for the filname, the class ignores all calls to its methods. This way, with no changes other than the constructor calls, I can control whether the XML files get written. My JackCompiler has options -x and -xt that cause it to generate the xxx.xml and xxxT.xml files. The compileXxx methods in the compiler look like this:
Compiles <class-var-dec> :=
('static' | 'field') <type> <var-name> (',' <var-name>)* ';'
ENTRY: Tokenizer positioned on the initial keyword.
EXIT: Tokenizer positioned after final ';'.
if (self.tokenizer.Keyword() == KW_STATIC):
variableKind = SYMK_STATIC
variableKind = SYMK_FIELD
("self" is Python's "this". _ExpectKeyword parses the next token and if it is not a keyword, or is not one of the keywords in the argument list, prints an error message and raises an exception. _CompileDec does all the remaining work for _CompileClassVarDec and _CompileVarDec.)