Tokenizing seems to include the delimiter in the token following a new delimiter

Question

The program in question is a phonebook application that takes formatted user input (e.g. ADD SampleName;SamplePhoneNumber;SampleCategory).

This method is supposed to separate this into four Strings:

• The command "ADD"
• Each of the other 3 tokens.

The first delimiter is a space, the other two are ;. When I use the following code, for some reason a space is included as a prefix to SampleName. I have no idea why this happens, or how to correct this in a practical fashion. I'm used to C++, and I'm just learning Java. Any suggestions are appreciated.

Here's the method:

public static Vector tokenize(String com)
{
   Scanner scanner = new Scanner(com);

Vector vs = new Vector();
String s;

while(scanner.hasNext())
{
    if(vs.size()==0)
    {
                scanner.useDelimiter("\\p{javaWhitespace}+");
                s = scanner.next();  // Sets the first delimiter to ' '
                scanner.useDelimiter("[;]");
    }
    else
    {
                scanner.useDelimiter("[;]");
                s = scanner.next();  // Sets all other delimiters as ';'
    }
    vs.add(s);  //  Adds the string s to the vector of strings vs
}

return vs;
}

Answer 1

It appears the extra spaces are kept once you switch delimiters. You could quite easily get around this problem by using the same delimiter throughout:

public static Vector tokenize(final String com) {
    Scanner scanner = new Scanner(com);
    scanner.useDelimiter("[;\\p{javaWhitespace}]+");
    Vector vs = new Vector();
    while (scanner.hasNext()) {
        vs.add(scanner.next()); // Adds the string to the vector of strings vs
    }
    return vs;
}

Answer 2

or this

public static Vector tokenize(final String com) {
    String[] tokens = com.split(" |;");
    Vector<String> vs = new Vector<String>(tokens.length);
    for (String s : tokens) {
            vs.add(s);
    }
    return vs;
}