Reducing memory usage while building a Zip file

Question

My application has a feature in which it exports data from the database. It builds a zip file containing the data using the Ionic ZIP library.

The process works by retrieving data in pages of 1,000 rows each. The data retrieved contains 2 images (JPEGS) per row of data. The "export" consists of an HTML file with an HTML table in it. The table contains important information from each row and links to the images. Each image is stored in the ZIP file as a JPEG file. The HTML written contains an anchor tag that links to the appropriate image file.

Everything works, but I recently found that an OutOfMemoryException is thrown when exporting large numbers of rows (over 24,000 in the case where I found the problem). In looking at my code, I don't see any obvious leaks (or I wouldn't be posting this, would I? ;) )

Here's the method that outputs the data being exported:

private void OutputReads( StreamWriter writer, ZipFile zipFile, BackgroundWorker worker, ExportArgs args ) {
    int ReadCount = Math.Min( args.ReadMatches, args.ResultsLimit );

    writer.WriteLine( "<h2>Matching Reads:</h2>" );
    writer.WriteLine( "<p>Number in Database: {0}</p>", args.ReadMatches.ToString( "#,##0" ) );
    writer.WriteLine( "<p>Number in Report:   {0}</p>",      ReadCount  .ToString( "#,##0" ) );
    writer.WriteLine();

    if ( args.ReadMatches == 0 ) {
        return;
    }

    writer.WriteLine( "<table border=\"1\" cellspacing=\"0\" bordercolordark=\"black\" bordercolorlight=\"black\">" );
    writer.WriteLine( "<tr>" );
    writer.WriteLine( "<th width=\"110\"><b>Plate</b></th>" );
    writer.WriteLine( "<th width=\"60\"><b>State</b></th>" );
    writer.WriteLine( "<th width=\"200\"><b>Date / Time</b></th>" );
    writer.WriteLine( "<th width=\"142\"><b>Latitude</b?</th> " );
    writer.WriteLine( "<th width=\"142\"><b>Longitude</b?</th> " );
    writer.WriteLine( "<th width=\"125\"><b>Alarm Class</b></th>" );
    writer.WriteLine( "<th width=\"150\"><b>Notes</b></th>" );
    writer.WriteLine( "<th width=\"150\"><b>Officer Notes</b></th>" );
    writer.WriteLine( "<th width=\"150\"><b>Images</b></th>" );
    writer.WriteLine( "</tr>" );

    int noPages = ReadCount / args.PageSize + ( ReadCount % args.PageSize == 0 ? 0 : 1 );

    for ( int pageNo = 0, count = 0; pageNo < noPages; pageNo++ ) {
        if ( worker.CancellationPending ) {
            return;
        }

        ReadViewModel[] reads = null;

        try {
            reads = DataInterface.GetReads( args.LocaleCode, args.Plate, 
                                            args.StartDate, args.EndDate, 
                                            args.AlarmClasses, args.HotListId, 
                                            args.PageSize, pageNo
                                          );

        } catch ( DataAccessException ex ) {
            . . .
        } catch ( ThreadAbortException ) {
            // The thread is stopping.  Stop processing now.

        } catch ( Exception ex ) {
            . . .
        }

        foreach ( ReadViewModel read in reads ) {
            if ( worker.CancellationPending ) {
                return;
            }

            writer.WriteLine( "<tr>" );
            writer.WriteLine( "<td width=\"110\"><p align=\"left\" style=\"text-align:left\">{0}</p></td>"    , read.Plate );
            writer.WriteLine( "<td width=\"60\" ><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.State );
            writer.WriteLine( "<td width=\"150\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.TimeStamp );
            writer.WriteLine( "<td width=\"125\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.GPSInformation == null ? "&nbsp;" : read.GPSInformation.Position.Latitude.ToString(  "##0.000000" ) );
            writer.WriteLine( "<td width=\"125\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.GPSInformation == null ? "&nbsp;" : read.GPSInformation.Position.Longitude.ToString( "##0.000000" ) );
            writer.WriteLine( "<td width=\"125\"><p align=\"center\" style=\"text-align:center\">{0}</p></td>", read.AlarmClass     == null ? "&nbsp;" : read.AlarmClass );
            writer.WriteLine( "<td width=\"150\"><p align=\"left\" style=\"text-align:left\">{0}</p></td>"    , read.       Notes   == null ? "&nbsp;" : read.       Notes );
            writer.WriteLine( "<td width=\"150\"><p align=\"left\" style=\"text-align:left\">{0}</p></td>"    , read.OfficerNotes   == null ? "&nbsp;" : read.OfficerNotes );

            if ( read.ImageData == null ) {
                try {
                    DataInterface.GetPlateImage( read );
                } catch ( DataAccessException ex ) {
                    . . .
                } catch ( Exception ex ) {
                    . . .
                }
            }

            if ( read.OverviewImages == null ) {
                try {
                    DataInterface.GetOverviewImages( read );
                } catch ( DataAccessException ex ) {
                    . . .
                } catch ( Exception ex ) {
                    . . .
                }
            }

            if ( read.ImageData != null ) {
                string ext = LPRCore.CarSystem.ImageDataAccessor.GetImageFileExtension( read.ImageData );
                writer.Write( "<td width=\"150\"><p align=\"left\" style=\"text-align:left\"><a href=\".\\Images\\{0}.{1}\" target=\"_blank\">BW</a>", read.ID.ToString( "N" ), ext );

                string fileName = string.Format( ".\\Images\\{0}{1}", read.ID.ToString( "N" ), ext );

                if ( !zipFile.ContainsEntry( fileName ) ) {
                    zipFile.AddEntry( fileName, read.ImageData );
                }
            } else {
                writer.Write( "No Plate Image" );
            }

            if ( read.OverviewImages != null && read.OverviewImages.Length > 0 ) {
                for ( int i = 0; i < read.OverviewImages.Length; i++ ) {
                    string ext = LPRCore.CarSystem.ImageDataAccessor.GetImageFileExtension( read.OverviewImages[ i ].ImageData );
                    writer.Write( " - <a href=\".\\Images\\{0}_C{1}{2}\" target=\"_blank\">Color {1}</a>", read.ID.ToString( "N" ), i == 0 ? string.Empty : i.ToString(), ext );

                    string fileName = string.Format( ".\\Images\\{0}_c{1}{2}", read.ID.ToString( "N" ), i == 0 ? string.Empty : i.ToString(), ext );
                    if ( !zipFile.ContainsEntry( fileName ) ) {
                        zipFile.AddEntry( fileName, read.OverviewImages[ i ].ImageData );
                    }
                }
            } else {
                writer.Write( "No Overview Images" );
            }

            writer.WriteLine( "</p></td>" );
            writer.WriteLine( "</tr>" );
            count++;
            worker.ReportProgress( count, args );
        }
    }
    writer.WriteLine( "</table>" );
    writer.WriteLine();
}

I'm thinking that maybe the Zip File is the problem as it isn't being flushed to disk or anything like that and just keeps getting bigger and bigger as rows are processed.

Is there a way that I can either flush the zip file to disk and release all of the images so the garbage collector will free them up or is there another way using this library to build the zip file using less memory?

Answer 1

I was able to come up with a fix for this with the help of another developer here.

The DotNetZip library defines an overload of the ZipFile.AddEntry method that takes a delegate of type WriteDelegate as a parameter. The delegate is passed the name of the file and a Stream to which the file's contents must be written as parameters.

I defined a method in my code called GetImageBytes:

private void GetImageBytes( string entryName, Stream stream ) {
    Guid imageId;
    Guid.TryParse( Path.GetFileName( entryName ).Substring( 0, 32 ), out imageId );
    using ( BinaryWriter writer = new BinaryWriter( stream ) ) {
        try {
            writer.Write( DataInterface.GetImageData( imageId ) );

        } catch ( DataAccessException ex ) {
            DbMonitor.HandleDatabaseError( ex );

        } catch ( Exception ex ) {
            . . .
        }
    }
}

To make this work, I had to encode the ID of the image in the database in the file name. The code parses the ID from the file name and then retrieves the image bytes from the database by calling a method in my Service layer. Finally, it creates a BinaryWriter and sends the image data to the Stream.

My program can now export over 24,000 rows of data and not throw an OutOfMemoryException.