Compression issue with large archive of files in DotNetZip

Question

Greetings....

I am writing a backup program in c# 3.5, using hte latest DotNetZip. The basics of the program is to be given a location on a server and the max size of a spanned zip file and go. From there it should traverse all the folder/files from the given location and add them to the archive, keeping the exact structure. It should also compress everything down to a reasonable amount. A given uncompressed collection of folders/files could easily be 10-25gb, with the created spanned files being limited to about 1gb each.

I have everything working (using DotNetZip). My only challenge is there is little to no compession actually happening. I chose to use the "AddDirectory" method for simplicity of code and just generally how well it seemed to fit my project. After reading around I am second guessing that decision.

1. Given the below code and the large amount of files in an archive, should I compress each file as it is added to the zip? or should the Adddirectory method provide about the same compression?

2. I have tried every level of compression offered by Ionic.Zlib.CompressionLevel and none seem to help. Should I think about using an outside compression algorithm and stream it into my DotNetZip file?

using (ZipFile zip = new ZipFile())  
{  
  zip.AddDirectory(root.FullName);  

  if (zipPassword.Length > 0)  
    zip.Password = zipPassword;  

  float size = zipGbSize * 1024 * 1024 * 1024;  

  zip.CompressionLevel = Ionic.Zlib.CompressionLevel.BestCompression;  
  zip.AddProgress += new EventHandler<AddProgressEventArgs>(Zip_AddProgress);  
  zip.ZipError += new EventHandler<ZipErrorEventArgs>(Zip_ZipError);  
  zip.Comment = "This zip was created at " + System.DateTime.Now.ToString("G");  
  zip.MaxOutputSegmentSize = (int)size;   //in gig  
  zip.Name = archiveDir.FullName + @"\Task_" + taskId.ToString() + ".zip";  
  zip.Save();  
}  

Thank you for any help!

Answer 1

1.Given the below code and the large amount of files in an archive, should I compress each file as it is added to the zip?

The way DotNetZip works is to compress each file as it is added to the archive. Your app does not need to do compression. DotNetZip does this for you.

or should the Adddirectory method provide about the same compression?

Entries added to a zip file via the AddDirectory() method go through the same code path when the zip archive is written, as entries added via AddFile(). The file data is compressed, then optionally encrypted, then written to the zip file.

---

an unsolicited tip: you don't need to do:

zip.AddProgress += new EventHandler<AddProgressEventArgs>(Zip_AddProgress);   

you can just do:

zip.AddProgress += Zip_AddProgress;   

---

how are you determining that no compression is occurring?

If you are curious about the compression on each entry, you can register a SaveProgress event handler. The SaveProgress event is fired at various times during the writing of an archive, including when saving begins, when DotNetZip begins writing the data for one entry, at various intervals during the writing of one entry, after finishing writing the data for each entry, and after finishing writing all data. These stages and described in the ZipProgressEventType enumeration. When the EventType is Saving_AfterWriteEntry, you can calculate the compression ratio for THAT particular entry.

To verify that compression is not occurring, I'd suggest that you register such a SaveProgress event and look at that compression ratio.

Also, as described above, some file types cannot be compressed. JPG, MPG, MP3, ZIP files, and others are not very compressible.

---

Finally, doing a backup may be lots easier to do if you just use the DotNetZip command-line tool. If all you want to do is backup a particular directory, you could use the command line tool (zipit.exe) and avoid writing a program. With the zipit.exe tool, if you use the -v option, the tool prints progress reports, and will display the compression for each entry, via the mechanism I described above. Even if you prefer to write your own program, you might consider using zipit.exe to verify that compression is, or is not, occuring when you use DotNetZip.

Answer 2

Im not sure to have understated your question, but the maximum size for any zip file its 4Gb. Maybe you have to create a new ZipFile every time you reach that limit.

Sorry if that doesnt help you.

Answer 3

What sort of data are you compressing? Some sorts of data just doesn't compress very well, for example JPEGs, or ZIP files which are already compressed.