C++ | How to format and export a simple binary file consisting of a 2-D matrix

Question

I need to export very large matrix, that can have various dimensions (100x10,1000000x4,100000x100) into a formatted binary file, which will carry the informations how columns are divided. This file is then loaded into Matlab for some plot results.

Until now, I was using simple ASCII export as follows in one of my function of class I made (file constructor defined elsewhere):

void file_io::writeColData(vector< vector<double> >* data){
    for(unsigned int i = 0;i < (*data).size();i++){
        for(unsigned int j = 0;j < (*data)[i].size();j++){
                file << (*data)[i][j] << '\t';
        }
        file << '\n';
    }
}

Which prints a nice file with full formating that is simply loaded into Matlab as load(file.txt);. Everything works fine, matrix is preserved, except that it is unbelievably and painfully slow.

Only thing I managed to get working in binary was an export of a simple vector:

void file_io::writeColDataBin(vector<double>* data){
    for(unsigned int i = 0;i < (*data).size();i++){
        file.write(reinterpret_cast<const char*>(&(*data)[i]), sizeof((*data)[i]));
    }
}

I can apply another dimension to void function above, however I cannot manage to format the output - all I get is just one long column of data (and I need more of those columns, number depending on situation).

I know that there are Matlab libraries which I can include (I am using Eclipse IDE), but I think it is just a big overkill for my needs, not mentioning the fact that I was not able to get it working after few painful hours. Another library I know of is MAT IO (matio.h), but I wasn't able to get this working either.

After the export, I need to import it into Matlab, possibly by

fid = ('data.dat','r','l');
data = fopen(fid,'double');

with (hopefully) showing the matrix I need.

Is there a way to accomplish this? A really... easy, simplistic way?

Thanks in advance!

Answer 1

So I finally made the output to satisfy my needs. The result is very similar to what iavr wrote (thank you for your quick response!), however I will copy my full (working) codes so it can be beneficial for someone else too.

This is mine function of class file_io that is writing the data:

void file_io::writeColDataBin(vector< vector<double> >* data){
    double rows = (double)(*data).size();
    double cols = (double)(*data)[0].size();
    file.write(reinterpret_cast<const char*>(&rows), sizeof(rows));
    file.write(reinterpret_cast<const char*>(&cols), sizeof(cols));
    for(unsigned int j = 0;j < (*data)[0].size();j++){
        for(unsigned int i = 0;i < (*data).size();i++){
            file.write(reinterpret_cast<const char*>(&(*data)[i][j]), sizeof((*data)[i][j]));
        }
    }
}

It simply writes the number of rows and columns first, then continues as normal, however writing the cells of matrix by rows first in every column, just then moving to next column. This is important, because Matlab is ordering its fread by columns, not rows. Size of rows and columns is also transformed into double, so Matlab can read the whole file at once.

The same class also opens a file of ofstream file as:

void file_io::fileOpenBin(const char* fileName){
    file.open(fileName, ios::out | ios::binary | ios::trunc);
}

After this, matrix is exported and loaded into Matlab with:

fid = fopen('data.dat','r');
data = fread(fid,'double');
fclose(fid);
rows = data(1);
cols = data(2);
data(1:2) = [];
data = reshape(data,rows,cols);

Rows and columns are imported, then first two cells are deleted from data and then they are reshaped into matrix needed.

Hope this helps someone in the future, it is probably not the quickest binary reading process, but it is definitely many times faster than reading ASCII.

Answer 2

Your attempt for binary output of array data looks ok so far. However, you need to also write the array dimensions (rows, columns) before the data. Then, assuming these numbers are written e.g. as uint64_t in C++, you can read the file in matlab as follows:

function matrix = load_2d(filename, data_type)

    fid = fopen(filename, 'rb');
    rows = fread(fid, 1, 'uint64');
    cols = fread(fid, 1, 'uint64');
    matrix = fread(fid, [rows cols], data_type);
    fclose(fid);

end

where data_type is a string representation of the matlab data type corresponding to your data type in C++.

Since your represent your matrix by a vector of vectors (with each inner vector holding a column), this will only work if all inner vectors are of the same size. Otherwise you'd need to write the size (rows) of each column individually and adjust load_2d accordingly. But if the target is a single 2d matrix, you'd have to truncate somehow.

Similarly, to save back:

function save_2d(filename, matrix, data_type)

    fid = fopen(filename, 'wb');
    [rows, cols] = size(matrix);
    fwrite(fid, rows, 'uint64');
    fwrite(fid, cols, 'uint64');
    fwrite(fid, matrix, data_type);
    fclose(fid);

end

I haven't tested though.