Parsing CSV input with a RegEx in java

Question

I know, now I have two problems. But I'm having fun!

I started with this advice not to try and split, but instead to match on what is an acceptable field, and expanded from there to this expression.

final Pattern pattern = Pattern.compile("\"([^\"]*)\"|(?<=,|^)([^,]*)(?=,|$)");

The expression looks like this without the annoying escaped quotes:

"([^"]*)"|(?<=,|^)([^,]*)(?=,|$)

This is working well for me - either it matches on "two quotes and whatever is between them", or "something between the start of the line or a comma and the end of the line or a comma". Iterating through the matches gets me all the fields, even if they are empty. For instance,

the quick, "brown, fox jumps", over, "the",,"lazy dog"

breaks down into

the quick
"brown, fox jumps"
over
"the"

"lazy dog"

Great! Now I want to drop the quotes, so I added the lookahead and lookbehind non-capturing groups like I was doing for the commas.

final Pattern pattern = Pattern.compile("(?<=\")([^\"]*)(?=\")|(?<=,|^)([^,]*)(?=,|$)");

again the expression is:

(?<=")([^"]*)(?=")|(?<=,|^)([^,]*)(?=,|$)

Instead of the desired result

the quick
brown, fox jumps
over
the

lazy dog

now I get this breakdown:

the quick
"brown
 fox jumps"
,over,
"the"
,,
"lazy dog"

What am I missing?

Answer 1

Operator precedence. Basically there is none. It's all left to right. So the or (|) is applying to the closing quote lookahead and the comma lookahead

Try:

(?:(?<=")([^"]*)(?="))|(?<=,|^)([^,]*)(?=,|$)

Answer 2

(?:^|,)\s*(?:(?:(?=")"([^"].*?)")|(?:(?!")(.*?)))(?=,|$)

This should do what you want.

Explanation:

(?:^|,)\s*

The pattern should start with a , or beginning of string. Also, ignore all whitespace at the beginning.

Lookahead and see if the rest starts with a quote

(?:(?=")"([^"].*?)")

If it does, then match non-greedily till next quote.

(?:(?!")(.*?))

If it does not begin with a quote, then match non-greedily till next comma or end of string.

(?=,|$)

The pattern should end with a comma or end of string.

Answer 3

When I started to understand what I had done wrong, I also started to understand how convoluted the lookarounds were making this. I finally realized that I didn't want all the matched text, I wanted specific groups inside of it. I ended up using something very similar to my original RegEx except that I didn't do a lookahead on the closing comma, which I think should be a little more efficient. Here is my final code.

package regex.parser;

import java.util.ArrayList;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class CSVParser {

    /*
     * This Pattern will match on either quoted text or text between commas, including
     * whitespace, and accounting for beginning and end of line.
     */
    private final Pattern csvPattern = Pattern.compile("\"([^\"]*)\"|(?<=,|^)([^,]*)(?:,|$)");  
    private ArrayList<String> allMatches = null;    
    private Matcher matcher = null;
    private String match = null;
    private int size;

    public CSVParser() {        
        allMatches = new ArrayList<String>();
        matcher = null;
        match = null;
    }

    public String[] parse(String csvLine) {
        matcher = csvPattern.matcher(csvLine);
        allMatches.clear();
        String match;
        while (matcher.find()) {
            match = matcher.group(1);
            if (match!=null) {
                allMatches.add(match);
            }
            else {
                allMatches.add(matcher.group(2));
            }
        }

        size = allMatches.size();       
        if (size > 0) {
            return allMatches.toArray(new String[size]);
        }
        else {
            return new String[0];
        }           
    }   

    public static void main(String[] args) {        
        String lineinput = "the quick,\"brown, fox jumps\",over,\"the\",,\"lazy dog\"";

        CSVParser myCSV = new CSVParser();
        System.out.println("Testing CSVParser with: \n " + lineinput);
        for (String s : myCSV.parse(lineinput)) {
            System.out.println(s);
        }
    }

}

Answer 4

I know this isn't what the OP wants, but for other readers, one of the String.replace methods could be used to strip the quotes from each element in the result array of the OPs current regex.