Behavior of strtok_s with sequential delimiters

Question

I parse 3 values ​​in parallel, which are separated by a specific separator.

token1 = strtok_s(str1, separator, &nextToken1); token2 = strtok_s(str2, separator, &nextToken2); token3 = strtok_s(str3, separator, &nextToken3); while ((token1 != NULL) && (token2 != NULL) && (token3 != NULL)) { //... token1 = strtok_s(NULL, separator, &nextToken1); token2 = strtok_s(NULL, separator, &nextToken2); token3 = strtok_s(NULL, separator, &nextToken3); } 

Suppose '-' is my delimiter. The behavior is that a string without consecutive delimiters:

 1-2-3-45 

will effectively result in each of these parts:

 1 2 3 45 

However, a line with two consecutive delimiters:

 1-2--3-45 

will not output a string of length 0, which is skipped so that the result:

 1 2 3 45 

but not

 1 2 3 45 

Which workaround or strategy is best suited to get all the actual parts, including zero-length? I would like to avoid reimplementing strtok_s if possible.

Answer 1

Unfortunately, strtok() ignores empty tokens. Even if you said you want to avoid this, there is no other way than to parse it yourself, using, for example, strchr() to find the next separator, and then copy the token to a temporary variable for processing. This way you can process empty tokens depending on what you like.

Answer 2

Yes, how this function works. This is more suitable for tasks such as parsing words, when several whitespace characters should not be considered as empty words.

I have disassembled many times. I will just write my own parser here, where the code checks one character at a time. It is not so difficult, and you can make it behave exactly as you need. For example, I posted some C ++ code to parse a CSV file in my article Reading and Writing CSV Files in MFC