Ruby StringScanner used for lexing: how to get line number?

Question

I use StringScanner for lexical analysis as follows:

def next @scanner.skip(/\s+/) value,kind=nil,nil TOKEN_DEF.each{|tok,regex| (kind=tok;break) if @scanner.scan(regex)} return Token.new(kind,value,@line,@scanner.pos) end 

In a first approximation, this works well, except that I cannot figure out how to get the @line number now.

I read a document where begin_of_line? the method seems appropriate, but I cannot figure out how to use it.

Answer 1

Save the text you scan in a variable and use "count"

In my code, I use the following:

 def current_line_number; @text[ [e-mail] ].count("\n") + 1; end 

Answer 2

This code seems not ready to go and, of course, somewhere an even more elegant solution, it just has to give you something to think about.

 class Retry < StandardError end class TextScanner def initialize(filename) @lines = IO.readlines(filename) @fiber = Fiber.new do @lines.each_with_index do |line, index| @scanner = StringScanner.new(line) @scanner.skip(/\s+/) value, kind = nil, nil begin got_token = false TOKEN_DEF.each do |tok, regex| if @scanner.scan(regex) Fiber.yield Token.new(tok, value, index, @scanner.pos) got_token = true end end raise Retry if got_token rescue Retry retry end end "fiber is finished" end end def next @fiber.resume end end text_scanner = TextScanner('sometextfile') puts text_scanner.next #=> first token puts text_scanner.next #=> second token puts text_scanner.next #=> third token ... puts text_scanner.next #=> "fiber is finished" 

Answer 3

I think I have a simple solution. Here he is:

 def next @line+=1 while @scanner.skip(/\n/) @line+=1 if @scanner.bol? @scanner.skip(/\s+/) @line+=1 if @scanner.bol? @scanner.skip(/\s+/) return :eof if @scanner.eos? TOKEN_DEF.each { |tok,syntax| (kind=tok;break) if @scanner.scan(syntax)} return Token.new(kind,nil,@line,@scanner.pos) end