OutOfMemoryException in Regex Matches when processing large files

Question

I've got an exception log from one of production code releases.

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.RegularExpressions.Match..ctor(Regex regex, Int32 capcount, String text, Int32 begpos, Int32 len, Int32 startpos)
   at System.Text.RegularExpressions.RegexRunner.InitMatch()
   at System.Text.RegularExpressions.RegexRunner.Scan(Regex regex, String text, Int32 textbeg, Int32 textend, Int32 textstart, Int32 prevlen, Boolean quick)
   at System.Text.RegularExpressions.Regex.Run(Boolean quick, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
   at System.Text.RegularExpressions.MatchCollection.GetMatch(Int32 i)
   at System.Text.RegularExpressions.MatchEnumerator.MoveNext()

The data it tries to process was about 800KB.

In my local tests it works perfectly fine. Have you ever seen similar behaviour, what can be the cause?

Shall I split the text before processing it, but obviously in that case regex might not match because the original file split from a random place.

My Regexes:

EDIT 2 :

I think this particular RegEx is causing the problem, when I test it out in an isolated environment it's eating the memory instantly.

((?:( |\.\.|\.|""|'|=)[\/|\?](?:[\w#!:\.\?\+=&@!$'~*,;\/\(\)\[\]\-]|%[0-9a-f]{2})*)( |\.|\.\.|""|'| ))?

EDIT

I was being wrong with my local test. I was loading up a big string then appending stuff to it which makes .NET Framework dizzy and then give an OOM exception during the RegEx instead of during string operations (or randomly, so ignore the previous stuff I've said).

This is a .NET Framework 2.0 application.

Answer 1

Without seeing your Regex, I don't know for sure but sometimes you can get problems like this because your matches are Greedy instead of Lazy.

The Regex engine has to store lots of information internally and Greedy matches can end up causing the Regex to select large sections of your 800k string, many times over.

There's some good information about this over here.

Answer 2

Based on your edit, it sounds like your code may be creating strings which take up large amounts of memory. This would mean that even though the out of memory exception is generated from within the Regex code, it's not actually because the Regex itself is taking up too much memory. Therefore, if using StringBuilder in your own code resolves the issue, then that's what you should do.

Answer 3

The first thing I would try, if it is possible for your application, would be to split up the input.

Would it be possible to read the file (if the input is a file) line-by-line, applying the Regular Expression that way?

You should take a look with CLR Profiler. It can take a little time to learn how to use, but it's worth it. It will help you visualize how much memory your objects use.