The second argument to Boost.Tokenizer's char_separator
is the kept_delims
parameter. It is used to specify a delimiters that will show up as tokens. The original code is specifying that "."
should be kept as a token. To resolve this, change:
ChSep sep(".", ".", boost::keep_empty_tokens);
to:
ChSep sep(".", "", boost::keep_empty_tokens);
// ^-- no delimiters will show up as tokens.
Here is a complete example:
#include <iostream>
#include <string>
#include <boost/foreach.hpp>
#include <boost/tokenizer.hpp>
int main()
{
std::string str = "This.is..a.test";
typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
boost::char_separator<char> sep(
".", // dropped delimiters
"", // kept delimiters
boost::keep_empty_tokens); // empty token policy
BOOST_FOREACH(std::string token, tokenizer(str, sep))
{
std::cout << "<" << token << "> ";
}
std::cout << std::endl;
}
Which produces the desired output:
<This> <is> <> <a> <test>