SSIS using too much memory to load large (40GB+) XML file into SQL Server table

Question

I need to load a single large (40GB+) XML file into an SQL Server 2012 database table using SSIS. I'm having problems because SSIS seems to be trying to load the entire document in memory instead of streaming it.

Here's more details of my SSIS package.

I've created an XML Source with the folowing properties:

• Data access mode: XML file from variable (but could be XML File Location)
• Variable name: variable that specifies the XML file path in my computer.
• XSD location: the path to the XSD that defines the XML being read.

The XML structure is simple, with only 3 hierarchical levels:

1. Root element with header information
2. One level defining collections of objects
3. The leaf level defining individual objects (each with a fixed set of fields)

I need to insert one database record per leaf element, repeating fields from the higher hierarchy levels. In other words, I need to flaten the XML hierarchy.

How can I make SSIS stream load the data, instead of trying to load the entire document in memory?

Answer 1

The XML source always loads the entire file. It uses XmlDocument to do so (last I checked).

The only thing you can do, is to split up the file somehow, then iteratively run each piece through your data flow.

Beyond that, you're looking at creating a custom data source, which is not trivial. It also represents a serious piece of code to maintain.

There may be third-party data sources which can do this. I had to write my own about five years ago.

Answer 2

Have you considered processing the files in smaller chunks?

I had the same issue before so I created a script component to process that 1 big XML file into 100's of smaller XML Files then do a forloop and iterate on all of the smaller XML Files to process.

To do this you cant use a StreamReader.ReadLine because it will still do the same thing, load that very large file so instead of that use System.IO.MemoryMappedFiles which is a designed class for this scenario.

Have a look here http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile.aspx