How to compress an in memory array to a *.zip archive?

Question

I have many multidimensional arrays like the one below. They are filled with specific values of course.

uint8_t data[32][64][32];

How can I compress such arrays in memory and then store them as *.zip archives to hard drive?

Answer 1

libzip is a ZIP compression and decompression library that allows one to operate on memory buffers as well as files.

Answer 2

You should tell us a bit more about your problem to allow for a specific answer. Do you need to move these files to a different machine? Are the dimensions of the arrays always the same?

In the simplest case where you just want to store the data - always with the same format - and only use it on your machine with the same code, all you need is a compression library. I have used http://www.quicklz.com/ before and it is very easy to integrate. In this case, the fact that it is a multidimensional array does not matter, neither does the type of the data. Just hand a pointer to the first element and the length to the compressor and save the result to a file (that you can name .zip if you like). Then when you want to reload it, read the file, hand it to the decompressor and cast the result into an array with the same dimensions.

For the more likely case where you have different types of arrays, and want them to be readable on other devices (and with other software) I suggest you use something like http://www.hdfgroup.org/HDF5/ that will allow you to store the data in a structured way including the datatype (with endianness etc.).

Answer 3

Take a look at libarchive (BSD license). It is very powerful and you have tight control over your data: you can combine different blocks of memory as "files" inside the compressed file (.tar.gz for example). There are several nice tutorials inside.