Is using strtok () in two functions safe?

Question

I use strtok() to parse strings in individual words and check them for something. If this thing is found, another function is called, which should also use strtok() .

I know strtok() not a re-contributor. Does this mean that if I call it in the second function, my position in the line of the first function will be lost? If so, would strtok() in the first function, and strtok_r() in the second would solve the problem? Is there any other solution?

edit: Thank you. it’s actually not possible to use strtok in two functions, but obviously strtok_r is not standard. redesign is ...

Answer 1

Q1:

Does this mean that if I call it in the second function, my position in the line in the first function will be lost?

A1: Yes, it is. If you do this, a new scan sequence will be launched with another line that you provided, and data for subsequent calls for your first line will be lost.

Q2:

If so, would strtok () be used in the first function, and strtok_r () in the second would solve the problem?

A2: The best approach would be to redesign your program design.

Q3:

Is there any other solution?

A3: If the cost of design changes is too high, I would suggest keeping a copy of the first line and a pointer containing the last token found. This may allow you to continue working from the last position (by getting the start pointer using strstr, for example) after you finish your second line.

Answer 2

Since strtok internally uses a global variable to hold how far it has moved in the string, mixing calls to strtok will fail, as you suspect. Your options:

• will switch to strtok_r , which has a similar API but is not standard C (this is on POSIX, though);
• avoid strtok in general in favor of some other function that does not carry a hidden global state, such as strsep (also not a standard);
• make sure your first function strtok out of strtok before calling another function that can call strtok .

In general, strtok is a function that is best avoided.

Answer 3

The strtok library strtok uses the internal static state for the current parsing position:

• when called with strings, it starts a new parsing,
• when called with NULL , it uses its internal state as the first argument.

If you directly or indirectly call strtok from the analysis loop, the internal state will be updated, and a call with NULL from the external region will not continue from the previous state, it may cause undefined behavior.

The Posix strtok_r function takes an explicit state argument, so it can be used in nested contexts. If this function is available on your system, use it in all places where you use strtok . Alternatively, you can use another method with strchr() or strcspn() .

strtok_r standardized in Posix. Depending on your target system, it may or may not be available. MacOS and most Unix systems are Posix compatible. Windows may have this name under a different name. If it is not available, you can override it in your program and conditionally compile it.

Here is a simple implementation that you use:

 char *strtok_r(char *s, const char *delim, char **context) { char *token = NULL; if (s == NULL) s = *context; /* skip initial delimiters */ s += strspn(s, delim); if (*s != '\0') { /* we have a token */ token = s; /* skip the token */ s += strcspn(s, delim); if (*s != '\0') { /* cut the string to terminate the token */ *s++ = '\0'; } } *context = s; return token; } 

Answer 4

As stated on the website http://en.cppreference.com/w/c/string/byte/strtok :

Each call to strtok changes a static variable: it is not thread safe.

So yes, you cannot call this function from two different functions at the same time (streaming), and you cannot call it like this:

 char input[] = "something that needs to be tokenize"; char *token = strtok(input, " "); while(token) { puts(token); anotherfunction(); token = strtok(NULL, " "); } void anotherfunction() { char input[] = "another string needs to be tokenize"; char *tok = strtok(input, " "); while(tok) { puts(tok); tok = strtok(NULL, " "); } }