how to improve this code about strtok_r

Question

I want to extract some tokens from a std::string by delimiter, and write a function as follows, but it still has errors:

    enum TO_TYPE { TO_INT=0, TO_FLOAT, TO_STRING};

    template<typename T>
    vector<T> string_to_token(string &str, const char* delim, TO_TYPE to_type)
    {
        char *p=new char[str.size()];
        memcpy(p,str.c_str(),str.size());
        switch(to_type)
        {
        case TO_INT:
        {
            vector<int> res;
            char *token;
            char *state;
            for (token = strtok_r(p, delim, &state);
                 token != NULL;
                 token = strtok_r(NULL, delim, &state))
            {
                res.push_back(atoi(token));
            }
            delete[] p;
            return res;
        }
            break;
        case TO_FLOAT:
        {
            vector<float> res;
            char *token;
            char *state;
            for (token = strtok_r(p, delim, &state);
                 token != NULL;
                 token = strtok_r(NULL, delim, &state))
            {
                res.push_back(atof(token));
            }
            delete[] p;
            return res;
        }
            break;
        case TO_STRING:
        {
            vector<string> res;
            char *token;
            char *state;
            for (token = strtok_r(p, delim, &state);
                 token != NULL;
                 token = strtok_r(NULL, delim, &state))
            {
                res.push_back(string(token));
            }
            delete[] p;
            return res;
        }
            break;
        }
    }

Usage:

    string str="lab,yuv,hsv";
    vector<string> colorspace=string_to_token<string>(str,",");

It has errors in the line return res.

I have tried to use void (*change_func)(char *temp_str) as the callback function, but I don't know how to implement it. If you were convenient, could you give me some advice, please? Thanks a lot.

Answer 1

The reason for the error is that your template will be specilized at compilation as (example given for int):

template<>
vector<int> string_to_token(string &str, const char* delim, TO_TYPE to_type)
{
    // ... 
    switch(to_type)
    {
    case TO_INT:
    {
        vector<int> res;
        // ...
        return res; // this is OK
    }
        break;
    case TO_FLOAT:
    {
        vector<float> res;
        // ...
        return res; // this will attempt to return vector<float> for a vector<int>
    }
        break;
    case TO_STRING:
    {
        vector<string> res;
        // ...
        return res; // this will attempt to return vector<string> for a vector<int>
    }
        break;
    }
}

You are essentially trying to solve two problems with this function:

• tokenization (see solutions here)

• string conversion.

Consider splitting this function in two parts:

• first, a non-templated function that tokenizes the string into a std::vector.

• second, a transformation (that can be easily templated by type):

Code:

std::vector<std::string> strings = tokenize(str, ","); // see link above
std::vector<int> ints{strings.size()};
std::transform(strings.begin(), strings.end(), ints.begin(),
    [](const std::string& s) { return std::stoi(s); });