A bit confused about the relationship between Tokenizer and Compilation

[riverfish]

In this week's project, it asks you to create the Tokenizer first that spits out initial XML code in the form of 'xxxT.xml'.

Then after the Tokenizer has successfully passed the tests, are you asked to complete the Compilation module which spits out the final version of the XML code in the form of 'xxx.xml' (without the T).


Does this mean that the Tokenizer will have a write function that creates the intiial xxxT.xml?

And then the Compilation module will take that xxxT.xml as an input and output the final xxx.xml file?

Because I had the impression that all the reading would be done by the Tokenizer and all the writing would be done by the Compilation.

[ivant]

N.B. I don't participate in the Coursera course in any way. I'm not aware if there are any additional requirements that it has.

The overall goal is to create a compiler, which takes a bunch of JACK files and produces the corresponding .vm files. This is a large and complex task, so it's split in several smaller ones. And you want to be able to test them. That's where these files come to play: the xxxT.xml files are used to test just the Tokenizer. Then the xxx.xml files are used to test the Parser (it should use the Tokenizer directly, not by reading the xxxT.xml files).

In the next assignments you'll start to produce real VM code, again directly, not by reading the xxx.xml files. But it's a good idea to write your tools in such a way that when they are run with some options they will still produce the XML files. This will help you debug problems in the future.

[cadet1620]

The xxxT.xml files are strictly for debugging the tokenizer. As soon as you are happy with your tokenizer you no longer need to write them. Coursera does not require you to generate them.

Note that when you convert JackAnalyzer into JackCompiler, it may be useful to (optionally) write the xxx.xml files in parallel with the xxx.vm files. Looking at the end of the XML file can help debugging when your compiler crashes deep into recursive parsing.

I wrote an XmlWriter class that takes a file name and used it for both the xxxT.xml and xxx.xml files. By passing in null for the filname, the class ignores all calls to its methods.  This way, with no changes other than the constructor calls, I can control whether the XML files get written.  My JackCompiler has options -x and -xt that cause it to generate the xxx.xml and xxxT.xml files.  The compileXxx methods in the compiler look like this:

    def _CompileClassVarDec(self):
        """
        Compiles <class-var-dec> :=
            ('static' | 'field') <type> <var-name> (',' <var-name>)* ';'

        ENTRY: Tokenizer positioned on the initial keyword.
        EXIT:  Tokenizer positioned after final ';'.
        """
        self._WriteXmlTag('<classVarDec>\n')
        self._ExpectKeyword((KW_STATIC, KW_FIELD))
        if (self.tokenizer.Keyword() == KW_STATIC):
            variableKind = SYMK_STATIC
        else:
            variableKind = SYMK_FIELD
        self._CompileDec(variableKind)
        self._WriteXmlTag('</classVarDec>\n')

("self" is Python's "this".  _ExpectKeyword parses the next token and if it is not a keyword, or is not one of the keywords in the argument list, prints an error message and raises an exception. _CompileDec does all the remaining work for _CompileClassVarDec and _CompileVarDec.)

--Mark

[cadet1620]

Note to self: don't post before the first cup of morning coffee... Ivan already mentioned writing the XML files in JackCompiler.

Enjoy the code sample.

--Mark

[riverfish]

cheers guys. much appreciated!