Using Ocamllex for Vocabulary Strings (Tiger Compiler)

Question

I am trying to follow Appel's "Modern Compiler Implementation in ML" and I am writing lexer using Ocamllex.

The specification suggests that a lexer return strings after translating escape sequences. The following code is an excerpt from the ocamllex input file:

rule tiger = parse ... | '"' { let buffer = Buffer.create 1 in STRING (stringl buffer lexbuf) } and stringl buffer = parse | '"' { Buffer.contents buffer } | "\\t" { Buffer.add_char buffer '\t'; stringl buffer lexbuf } | "\\n" { Buffer.add_char buffer '\n'; stringl buffer lexbuf } | "\\n" { Buffer.add_char buffer '\n'; stringl buffer lexbuf } | '\\' '"' { Buffer.add_char buffer '"'; stringl buffer lexbuf } | '\\' '\\' { Buffer.add_char buffer '\\'; stringl buffer lexbuf } | eof { raise End_of_file } | _ as char { Buffer.add_char buffer char; stringl buffer lexbuf } 

Is there a better way?

Answer 1

You may be interested to see how Ocaml lexer (search and string ) does it. In essence, this is the same method as yours, without a good local buffer (I find your code nicer at this point, but it is a little less efficient), a little more complicated because it supports more screens and uses an escape table (char_for_backslash) to factorize similar rules.

Also, you repeat the "\\n" rule twice, and I think that 1 is a very pessimistic estimate of your string length, I would prefer to use 20 here (to avoid unnecessary resizing).