java StringTokenizer unexpected results

Question

i have the following code which will tokenize the string to create list of Objects:

import java.util.StringTokenizer;


public class TestStringTokenizer {
    private final static String INTERNAL_DELIMETER = "#,#";
    private final static String EXTERNAL_DELIMETER = "#|#";
    public static void main(String[]aregs){
        String test= "1#,#Jon#,#176#|#2#,#Jack#,#200#|#3#,#Jimmy#,#160";
        StringTokenizer tokenizer = new StringTokenizer(test, EXTERNAL_DELIMETER);
        while(tokenizer.hasMoreElements()){
            System.out.println(tokenizer.nextElement());
            //later will take this token and extract elements
        }
    }
}

What i expected output was
1#,#Jon#,#176
2#,#Jack#,#200
3#,#Jimmy#,#160

What i got was : 1
,
Jon
,
176
2
,
Jack
,
200
3
,
Jimmy
,
160

if i change the internal delimeter to some thing like , it will work properly why is this behavior happening ?

Answer 1

StrinTokenizer constructor's second parameter is delimiters(Each character is a delimiter)

You can use String.split instead

public class TestStringTokenizer {
    private final static String INTERNAL_DELIMETER = "#,#";
    private final static String EXTERNAL_DELIMETER = "#|#";
    public static void main(String[]aregs){
        String test= "1#,#Jon#,#176#|#2#,#Jack#,#200#|#3#,#Jimmy#,#160";
        for (String s : test.split("#\\|#"))
            System.out.println(s);
        }
    }
}

Answer 2

AccordIng to the StringTokenizer JavaDocs

http://docs.oracle.com/javase/7/docs/api/java/util/StringTokenizer.html

StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.

Use String.split instead:

String[] strArr = stringToSplit.split(INTERNAL_DELIMETER);

The only change you need to make is that the or-pipe ("|") in EXTERNAL_DELIMETER is a special regular expression character, and must be escaped: "\\|".

More information can be found in the String.split Javadoc:

http://docs.oracle.com/javase/7/docs/api/java/lang/String.html#split(java.lang.String)

Answer 3

StrinTokenizer can't work with expression as a delimiter, try Scanner instead

    Scanner sc = new Scanner(test);
    sc.useDelimiter("#\\|#");
    while (sc.hasNext()) {
        System.out.println(sc.next());
    }