Stackoverflow in pattern matching in java

Question 1

Why does StackOverflowError occur?

In reference implementation of Pattern class (which comes with Oracle's JRE, OpenJDK, and a number of other JVMs), greedy and lazy quantifiers are implemented with recursion¹ when the repeated pattern is non-trivial. Therefore, you will run into StackOverflowError when the input string is long enough.

^{¹ Recursion is a quick but not scalable solution to allow backtracking in the pattern. Better implementation uses a data structure to store backtracking points
(which basically converts recursive solution to iterative solution with stack).}

Solution

The following regex should work:

"(?:\"(?:[^\"\\\\]++|\\\\.)*+\"|[^ \"]++)++"

Well, the regex is quite confusing with 2 layers of escaping: escaping in Java string literal and escaping in regex syntax.

The raw regex when you print the string out. My explanation will be based on the raw regex.

(?:"(?:[^"\\]++|\\.)*+"|[^ "]++)++

Explanation

Since you only care about what the whole regex matches, all the capturing groups (pattern) has been turned into non-capturing group (?:pattern) for efficiency.

The first alternative "(?:[^"\\]++|\\.)*+" matches a quoted string.

The second alternative [^ "]++ matches a sequence of character that doesn't contain space and double quote ".

(?:
   "             # Double quote
   (?:
      [^"\\]++   # A sequence of characters that are not " and \
      |          # OR
      \\.        # Escape sequence: \ followed by any character (except line terminators)
   )*+           # Match 0 or more of the sequences above (allows empty string)
   "             # Double quote
   |
   [^ "]++
)++

Since the regex is written so that there is no needs for backtracking, all quantifiers are made possessive. Since Pattern class implements possessive quantifier with a loop, instead of recursion as the case with greedy/lazy quantifiers, StackOverflowError will not occur.

I remove the need for backtracking by writing the regex so that it matches the correct string on first try:

Since [^"\\] excludes the \, there is no way we can "steal" a \ from an escaping sequence, or "steal" a " and mess up the closing quote, we can safely advance ahead without backtracking. This explains the possessive quantifier here [^"\\]++. There is no need to assign a quantifier here, but I do this to reduce the work on the branching.
Since both [^"\\]++ and \\. can't "steal" a " and mess up the closing quote, we can advance ahead without backtracking. This explains the possessive quantifier here (?:[^"\\]++|\\.)*+
[^ "] can't start a quoted string, and it also can't match a space (delimiter). This is why we can use possessive quantifier on it.
Since "(?:[^"\\]++|\\.)*+" and [^ "]++ can't mess up the match of each other, we can make the outer most quantifier possessive.

Example of a regex that doesn't match things correctly on first try and only get the correct result after backtracking would be ^([bcd]+:[ab]+)+$ against inputs such as b:ab:a. The first iteration will match b:ab, which cause the 2nd iteration to fail, then it backtracks and retry with the first iteration being b:a and then successfully match the whole string.

Question 2

Your regex is broken:

(([\"]([^\\\"]|\\.)+[\"]|[^ ]+))+
  #### ####### ###
  |    |       ---------------- A dot
  |    ------------------------ Any character not "
  ----------------------------- A " (no need to put it in a character class)

At this point I stopped looking further, because I am sure this is not what you want.

By the way, I recommend writing the regex first and only then do the quoting (you could write yourself a tool that does this, it is purely mechanical: add one \ before every " and every \, then enclose in ""). Also, don't use character classes for single characters.

In fact, it appears what you're looking for are words, or strings. So, why don't you say just that.

You can use a top down approach:

REGEX = (WORD|STRING)
WORD  = \w+                  -- or \p{L} or something like that
STRING = "(SOMETHING)*"
SOMETHING = \\["\\]|[^\\"]    -- an escaped quote, an escaped backslash or 
                              -- something that is neither a backslash nor a quote

Now:

Replace the uppercase words by their right hand sides
Remove unnecessary (), if any
Quote

You can test important sub-regexes separately, for example the STRING. Turned out I had several errors in my first version, and this even when writing unquoted! To write/discuss such a regex in the form java demands from the start is virtually impossible.

Question 3

Thanks for all your responses. Atlast i found my mistake. The actual reason is for stackoverflow is not my regex. My regex was correct. I used eclipse for coding. The actual reason for stackoverflow is my stack size. Intially my stacksize was 1Mb. I increased my stack size for the program in my eclipse and there was no error.

Java stack overflow error - how to increase the stack size in Eclipse?

UPDATE:

There is no need to change the stack size. As mentioned by nhahtdh, I have changed the regex to regex with possessive quantifier and there was no stackoverflow error.

My Regex is now ("([^\\"]|\\.)++"|[^\s]++)

To learn more about Possessive quantifier follow this link.

Question 4

The first thing to try is to increase the stack size.

If that does not work, you might have hit a bug. You could try a different JVM and set up the JVM to use something other than OpenJDK for the class library, and tinker with the regex to see exactly what is triggering it.