Is strtok() useless?

Question

php.net/strtok

Example straight from the docs:

$tok = strtok($string, " \n\t");

while ($tok !== false) {
    echo "Word=$tok<br />";
    $tok = strtok(" \n\t");
}

In the 2nd argument of strtok() you can specify the characters that are used for splitting. If a specify multiple characters, how do I know by which one did strtok do the split?

If I have no way of find that out, does't that make this function completely useless for creating parsers, lexers etc. ?

Answer 1

Is strtok() useless - no.

Is it useful for creating parsers/lexers etc. - not really.

If you are parsing something you have a lot more of a problem than just splitting a string based on some delimiter or other - e.g. handling of embedded "a string" or 'other "string"' means that the meaning of a delimiter is dependent on context.

Answer 2

As we can see in the docs:

Note that only the first call to strtok uses the string argument. Every subsequent call to strtok only needs the token to use, as it keeps track of where it is in the current string.

This function is meant to tokenize a string in a secuencial basis.

Whereas explode returns an array with all the string split up, this function, in every call, returns the first token from the string, using the delimiter. In the next call, it returns the next token...but you can change the delimiter! so you can iterate over a string, and extract parts following a more complex logic, maybe evaluating the previous token, or obtaining the delimiter in runtime, etc

Of course you can write a parser, as well as you can do it with many of the other functions that you know, but this is only a very initial step, you would have to write all the logic by yourself (for instance, if you find a { probably you can tokenize until } but then you will have to recursively parse the returned string, and so on)