Followup: Error: Failed to parse regular expression “”: pattern too large - compile failed

Question

This is a followup on Error: Failed to parse regular expression "": pattern too large - compile failed...

I am hitting this problem again.

Can anyone please please explain how BigQuery allocates resources for REGEXP functions? Is there a known issue (and ways to avoid it) of "too many"\"too complex" REGEXP expressions in a single BQ query?

Many Thanks

Answer 1

Regular expression parsing has some unfortunate corner cases that can eat up a lot of processing cycles (see http://www.regular-expressions.info/catastrophic.html). The regular expression state machines can also eat up a lot of memory. Because of this, BigQuery has a budget for regular expressions. The current value is something like 2MB per query. This is a guideline, not a published guarantee, however, and is subject to change at any time.

BigQuery is documented to use Re2 (code here: https://code.google.com/p/re2/) for regular expression parsing; You can try running your regular expressions with this library using a budget of 2MB and check whether they succeed or not.