How to catch a list of tokens in an antlr3 tree grammar?

Question

I took a dummy language, for example: It just accepts one or more "!" . its lexers and grammar rules:

grammar Ns; options { output=AST; ASTLabelType=CommonTree; } tokens { NOTS; } @header { package test; } @lexer::header { package test; } ns : NOT+ EOF -> ^(NOTS NOT+); NOT : '!'; 

ok, as you can see, this is a language that accepts "!" or '!!!' or '!!!!!'...

and I defined some meaningful classes for building AST:

 public class Not { public static final Not SINGLETON = new Not(); private Not() { } } public class Ns { private List<Not> nots; public Ns(String nots) { this.nots = new ArrayList<Not>(); for (int i = 0; i < nots.length(); i++) { this.nots.add(Not.SINGLETON); } } public String toString() { String ret = ""; for (int i = 0; i < this.nots.size(); i++) { ret += "!"; } return ret; } } 

and here is the grammar of the tree:

 tree grammar NsTreeWalker; options { output = AST; tokenVocab = Ns; ASTLabelType = CommonTree; } @header { package test; } ns returns [Ns ret] : ^(NOTS n=NOT+) {$ret = new Ns($n.text);}; 

and the main class code with some sample data for testing the generated classes:

 public class Test { public static void main(String[] args) throws Exception { ANTLRInputStream input = new ANTLRInputStream(new ByteArrayInputStream("!!!".getBytes("utf-8"))); NsLexer lexer = new NsLexer(input); CommonTokenStream tokens = new CommonTokenStream(lexer); NsParser parser = new NsParser(tokens); CommonTree root = (CommonTree) parser.ns().getTree(); NsTreeWalker walker = new NsTreeWalker(new CommonTreeNodeStream(root)); try { NsTreeWalker.ns_return r = walker.ns(); System.out.println(r.ret); } catch (RecognitionException e) { e.printStackTrace(); } } } 

but the final print result is '!', except for the expected '!!!'. which is mainly because this line of code:

 ns returns [Ns ret] : ^(NOTS n=NOT+) {$ret = new Ns($n.text);}; 

$ n above is captured by only one "!" I don’t know how to capture all three tokens! in other words, a list of "!" with $ n. Can anybody help? Thanks!

Answer 1

The fact that only one is printed ! , is that your rule:

 ns returns [Ns ret] : ^(NOTS n=NOT+) {$ret = new Ns($n.text);} ; 

gets more or less translated as:

 Token n = null LOOP n = match NOT_token END return new Ns(n.text) 

For this, n.text will always be only one ! .

What you need to do is collect these NOT tags in the list. In ANTLR, you can create a list of tokens using the += operator instead of the "single token" = operator. Therefore, change the ns rule to:

 ns returns [Ns ret] : ^(NOTS n+=NOT+) {$ret = new Ns($n);} ; 

which translates as:

 List n = null LOOP n.add(match NOT_token) END return new Ns(n) 

Be sure to change the constructor of your ns class instead of List :

 public Ns(List nots) { this.nots = new ArrayList<Not>(); for (Object o : nots) { this.nots.add(Not.SINGLETON); } } 

after which the output of your test class will be as follows:

 !!! 

Good luck