Entering a string in flex lexer

Question

I want to create a read-eval-print loop using the flex / bison parser. The problem is that the generated flexer lexer wants input of type FILE *, and I would like it to be char *. Is there any way to do this?

It was proposed to create a channel, pass it to a string and open a file descriptor and send to lexer. It's pretty simple, but it seems confusing and not very platform independent. Is there a better way?

Answer 1

The following procedures are available to configure input buffers for scanning lines in memory instead of files (as yy_create_buffer does):

• YY_BUFFER_STATE yy_scan_string(const char *str) : scans a string with a terminating NUL`
• YY_BUFFER_STATE yy_scan_bytes(const char *bytes, int len) : scans len bytes (including, possibly, NUL), starting from the location byte

Note that both of these functions create, return the corresponding YY_BUFFER_STATE descriptor (which you must delete with yy_delete_buffer () when you're done with it), so yylex () scans a copy of the string or bytes. This behavior may be desirable since yylex () changes the contents of the buffer that it scans).

If you want to avoid copying (and yy_delete_buffer) using:

• YY_BUFFER_STATE yy_scan_buffer(char *base, yy_size_t size)

main sample:

 int main() { yy_scan_buffer("a test string"); yylex(); } 

Answer 2

See this section of the Flex manual for information on how to scan buffers in memory, such as strings.

Answer 3

Here is what I need to do:

 extern yy_buffer_state; typedef yy_buffer_state *YY_BUFFER_STATE; extern int yyparse(); extern YY_BUFFER_STATE yy_scan_buffer(char *, size_t); int main(int argc, char** argv) { char tstr[] = "line i want to parse\n\0\0"; // note yy_scan_buffer is is looking for a double null string yy_scan_buffer(tstr, sizeof(tstr)); yy_parse(); return 0; } 

you cannot violate typedef, which makes sense when you think about it.

Answer 4

flex can parse char * using any of three functions: yy_scan_string() , yy_scan_buffer() and yy_scan_bytes() (see documentation ). Here is an example of the first:

 typedef struct yy_buffer_state * YY_BUFFER_STATE; extern int yyparse(); extern YY_BUFFER_STATE yy_scan_string(char * str); extern void yy_delete_buffer(YY_BUFFER_STATE buffer); int main(){ char string[] = "String to be parsed."; YY_BUFFER_STATE buffer = yy_scan_string(string); yyparse(); yy_delete_buffer(buffer); return 0; } 

Equivalent operators for yy_scan_buffer() (which require a null-terminated string):

 char string[] = "String to be parsed.\0"; YY_BUFFER_STATE buffer = yy_scan_buffer(string, sizeof(string)); 

My answer repeats the information provided by @dfa and @jlholland, but none of them work for me.

Answer 5

The accepted answer is incorrect. This will lead to memory leaks.

Internally, yy_scan_string calls yy_scan_bytes, which in turn calls yy_scan_buffer.

yy_scan_bytes allocates memory for a COPY of the input buffer.

yy_scan_buffer works directly with the supplied buffer.

With all three forms, you MUST call yy_delete_buffer to release buffer status information (YY_BUFFER_STATE).

However, with yy_scan_buffer you avoid internal allocation / copy / free internal buffer.

The prototype yy_scan_buffer does NOT accept const char *, and you SHOULD NOT expect the contents to remain unchanged.

If you have allocated memory to hold your string, you are responsible for freeing it AFTER you call yy_delete_buffer.

Also, don't forget that yywrap returns 1 (nonzero) when you parse ONLY this line.

The following is an example of COMPLETE.

 %% <<EOF>> return 0; . return 1; %% int yywrap() { return (1); } int main(int argc, const char* const argv[]) { FILE* fileHandle = fopen(argv[1], "rb"); if (fileHandle == NULL) { perror("fopen"); return (EXIT_FAILURE); } fseek(fileHandle, 0, SEEK_END); long fileSize = ftell(fileHandle); fseek(fileHandle, 0, SEEK_SET); // When using yy_scan_bytes, do not add 2 here ... char *string = malloc(fileSize + 2); fread(string, fileSize, sizeof(char), fileHandle); fclose(fileHandle); // Add the two NUL terminators, required by flex. // Omit this for yy_scan_bytes(), which allocates, copies and // apends these for us. string[fileSize] = '\0'; string[fileSize + 1] = '\0'; // Our input file may contain NULs ('\0') so we MUST use // yy_scan_buffer() or yy_scan_bytes(). For a normal C (NUL- // terminated) string, we are better off using yy_scan_string() and // letting flex manage making a copy of it so the original may be a // const char (ie, literal) string. YY_BUFFER_STATE buffer = yy_scan_buffer(string, fileSize + 2); // This is a flex source file, for yacc/bison call yyparse() // here instead ... int token; do { token = yylex(); // MAY modify the contents of the 'string'. } while (token != 0); // After flex is done, tell it to release the memory it allocated. yy_delete_buffer(buffer); // And now we can release our (now dirty) buffer. free(string); return (EXIT_SUCCESS); } 

Answer 6

In another mode, you can override the YY_INPUT function in the lex file, and then set the line to the LEX input. As below:

 #undef YY_INPUT #define YY_INPUT(buf) (my_yyinput(buf)) char my_buf[20]; void set_lexbuf(char *org_str) { strcpy(my_buf, org_str); } void my_yyinput (char *buf) { strcpy(buf, my_buf); } 

In your main.c, before scanning you need to set the lex buffer first:

 set_lexbuf(your_string); scanning... 

Answer 7

here is a small example of using bison / flex as a parser inside your cpp code to parse a string and change the string value to fit it (several parts of the code have been removed, so there may be irrelevant parts). parser.y:

 %{ #include "parser.h" #include "lex.h" #include <math.h> #include <fstream> #include <iostream> #include <string> #include <vector> using namespace std; int yyerror(yyscan_t scanner, string result, const char *s){ (void)scanner; std::cout << "yyerror : " << *s << " - " << s << std::endl; return 1; } %} %code requires{ #define YY_TYPEDEF_YY_SCANNER_T typedef void * yyscan_t; #define YYERROR_VERBOSE 0 #define YYMAXDEPTH 65536*1024 #include <math.h> #include <fstream> #include <iostream> #include <string> #include <vector> } %output "parser.cpp" %defines "parser.h" %define api.pure full %lex-param{ yyscan_t scanner } %parse-param{ yyscan_t scanner } {std::string & result} %union { std::string * sval; } %token TOKEN_ID TOKEN_ERROR TOKEN_OB TOKEN_CB TOKEN_AND TOKEN_XOR TOKEN_OR TOKEN_NOT %type <sval> TOKEN_ID expression unary_expression binary_expression %left BINARY_PRIO %left UNARY_PRIO %% top: expression {result = *$1;} ; expression: TOKEN_ID {$$=$1; } | TOKEN_OB expression TOKEN_CB {$$=$2;} | binary_expression {$$=$1;} | unary_expression {$$=$1;} ; unary_expression: TOKEN_NOT expression %prec UNARY_PRIO {result = " (NOT " + *$2 + " ) " ; $$ = &result;} ; binary_expression: expression expression %prec BINARY_PRIO {result = " ( " + *$1+ " AND " + *$2 + " ) "; $$ = &result;} | expression TOKEN_AND expression %prec BINARY_PRIO {result = " ( " + *$1+ " AND " + *$3 + " ) "; $$ = &result;} | expression TOKEN_OR expression %prec BINARY_PRIO {result = " ( " + *$1 + " OR " + *$3 + " ) "; $$ = &result;} | expression TOKEN_XOR expression %prec BINARY_PRIO {result = " ( " + *$1 + " XOR " + *$3 + " ) "; $$ = &result;} ; %% lexer.l : %{ #include <string> #include "parser.h" %} %option outfile="lex.cpp" header-file="lex.h" %option noyywrap never-interactive %option reentrant %option bison-bridge %top{ /* This code goes at the "top" of the generated file. */ #include <stdint.h> } id ([a-zA-Z][a-zA-Z0-9]*)+ white [ \t\r] newline [\n] %% {id} { yylval->sval = new std::string(yytext); return TOKEN_ID; } "(" {return TOKEN_OB;} ")" {return TOKEN_CB;} "*" {return TOKEN_AND;} "^" {return TOKEN_XOR;} "+" {return TOKEN_OR;} "!" {return TOKEN_NOT;} {white}; // ignore white spaces {newline}; . { return TOKEN_ERROR; } %% usage : void parse(std::string& function) { string result = ""; yyscan_t scanner; yylex_init_extra(NULL, &scanner); YY_BUFFER_STATE state = yy_scan_string(function.c_str() , scanner); yyparse(scanner,result); yy_delete_buffer(state, scanner); yylex_destroy(scanner); function = " " + result + " "; } makefile: parser.h parser.cpp: parser.y @ /usr/local/bison/2.7.91/bin/bison -y -d parser.y lex.h lex.cpp: lexer.l @ /usr/local/flex/2.5.39/bin/flex lexer.l clean: - \rm -f *.o parser.h parser.cpp lex.h lex.cpp