Exporting hundreds of thousands of records with ColdFusion

Question

Using ColdFusion 9.0.1, I need to export hundreds of thousands of database records to Excel XLSX or CSV (XLSX is preferred). This must be done on demand. So far I've tried using cfspreadsheet but it chokes when exporting more than a couple thousand rows in the XLSX format. However, exporting to XLS works fine (of course there is a ~65,000 row limit).

What are my options to export so many records? Theoretically the users could need to export as many as one million records. I'm also using SQL Server 2008 R2 -- is there a way to somehow export the records to a file there and then send the file through CF to the user? What options do I have? Thanks.

Answer 1

Since you are using SQL Server 2008, you could take advantage of SQL Server Reporting Services (SSRS) and create a report that can be called via web service (or HTTP GET/POST) by ColdFusion. SSRS has the capability to export reports as Excel as well. You'll need to read up on SSRS to make this work, but it's fairly easy to do.

Answer 2

As you've discovered, doing this with ColdFusion's <cfspreadsheet/> tag fails because it builds the entire document in memory, which leads to JVM OutOfMemory errors. What you need is something that buffers output to disk so you don't run out of memory. This suggests CSV, which is far easier to buffer. I imagine there are ways to do it with Excel as well, but I don't know them.

So two options for you:

1. use a Java library
2. use ColdFusion's fileOpen(), fileWrite(), fileClose() methods

I'll cover each in turn.

Java Libary

opencsv is my preference. This assumes of course you know how to setup a .jar on the ColdFusion classpath. If you do then it's a matter using its APIs to open a file and specify data for each line. It's really quite simple. Check its docs for examples.

ColdFusion Methods

Be forewarned there be dragons here.

If you are exporting numbers or strings that do not contain any double quotes or commas you can probably do this. If not, figuring out what to escape and how is why you use a library in the first place. Code is roughly as such:

<!--- query to get whatever data you're working with --->

<cfset csvFile = fileOpen(filePath, 'read')>
<cfloop query="yourQuery">
    <cfset csvRow = ""><!--- construct a csv row here from the query row --->
    <cfset fileWrite(csvFile, csvRow)>
</cfloop>

<cfset fileClose(csvFile)>

If the query data you're working with is also large you may be dealing with a nested loop to chunk it out.

Answer 3

Dustin, I had to investigate this myself, and as of this writing (Summer 2011), POI does a fine job of generating large files, but you have to use xlsx. The 3.8 beta source ships with an example named "BigGridDemo" which generates a 100K, 4-column workbook very quickly. I modified it to generate 300K, 125-column sheet, and it handled it in about 2 minutes. It created a 1.6 GB, 3.6-million-row workbook in a little over half an hour.

Granted, the code isn't the prettiest to look at, but it works. I suspect it'll pretty up a bit when ported to ColdFusion.