How can I read the MNIST dataset with C++?

Question

I was reading the following question:

How to read MNIST data in C++?

and there was some C++ code for reading the MNIST database. Upon trying it I found out that it worked fine until the place where it started reading the data.

which is the following code:

 for(int i=0;i<number_of_images;++i)
      {
        for(int r=0;r<n_rows;++r)
        {
            for(int c=0;c<n_cols;++c)
            {
                unsigned char temp=0;
                file.read((char*)&temp,sizeof(temp));
                //cout<<(int)temp<<" "; //printing the pixel in integer format

            }
        }
    }

I tried printing out the integer value of the variable "temp" however I didn't get the correct number for the pixels(all of them were zero). I'm not sure what's wrong there, each pixel takes one bytes space and then I convert it to an int and it doesn't work. Why does this happen? thank you in advance

Answer 1

When working with the MNIST data set, I had the same problem that you had. I could read the labels, but the training and test set images were mostly bogus; the training set was filled almost entirely with 175, and the testing set was filled almost entirely with 0s (except for the first 6 images). Rebooting did not fix the problem and I was unable to determine why the file reading was not working correctly.

For anyone with this same problem, I would suggest instead using the data files located at http://cis.jhu.edu/~sachin/digit/digit.html. The data is already organized by number (no label/image association required), and the arrays of pixel values are simply encoded one after the other. Knowing that each array is 28x28 and that there are 1000 images for each number, you can easily write code to input the individual image arrays of pixel values.

Answer 2

I had the same problem, the first few bytes of data being fine and then the rest was filled with 0s. It turns out that the file stream was failing when it first encountered byte 0x01A. The fix for this was changing:

ifstream file ("data");

to

ifstream file ("data", std::ios::binary);

Answer 3

Below is complete code to read mnist and convert it to cv Mat:

uint32_t swap_endian(uint32_t val) {
    val = ((val << 8) & 0xFF00FF00) | ((val >> 8) & 0xFF00FF);
    return (val << 16) | (val >> 16);
}

void read_mnist_cv(const char* image_filename, const char* label_filename){
    // Open files
    std::ifstream image_file(image_filename, std::ios::in | std::ios::binary);
    std::ifstream label_file(label_filename, std::ios::in | std::ios::binary);

    // Read the magic and the meta data
    uint32_t magic;
    uint32_t num_items;
    uint32_t num_labels;
    uint32_t rows;
    uint32_t cols;

    image_file.read(reinterpret_cast<char*>(&magic), 4);
    magic = swap_endian(magic);
    if(magic != 2051){
        cout<<"Incorrect image file magic: "<<magic<<endl;
        return;
    }

    label_file.read(reinterpret_cast<char*>(&magic), 4);
    magic = swap_endian(magic);
    if(magic != 2049){
        cout<<"Incorrect image file magic: "<<magic<<endl;
        return;
    }

    image_file.read(reinterpret_cast<char*>(&num_items), 4);
    num_items = swap_endian(num_items);
    label_file.read(reinterpret_cast<char*>(&num_labels), 4);
    num_labels = swap_endian(num_labels);
    if(num_items != num_labels){
        cout<<"image file nums should equal to label num"<<endl;
        return;
    }

    image_file.read(reinterpret_cast<char*>(&rows), 4);
    rows = swap_endian(rows);
    image_file.read(reinterpret_cast<char*>(&cols), 4);
    cols = swap_endian(cols);

    cout<<"image and label num is: "<<num_items<<endl;
    cout<<"image rows: "<<rows<<", cols: "<<cols<<endl;

    char label;
    char* pixels = new char[rows * cols];

    for (int item_id = 0; item_id < num_items; ++item_id) {
        // read image pixel
        image_file.read(pixels, rows * cols);
        // read label
        label_file.read(&label, 1);

        string sLabel = std::to_string(int(label));
        cout<<"lable is: "<<sLabel<<endl;
        // convert it to cv Mat, and show it
        cv::Mat image_tmp(rows,cols,CV_8UC1,pixels);
        // resize bigger for showing
        cv::resize(image_tmp, image_tmp, cv::Size(100, 100));
        cv::imshow(sLabel, image_tmp);
        cv::waitKey(0);
    }

    delete[] pixels;
}

Usage:

string base_dir = "/home/xy/caffe-master/data/mnist/";
string img_path = base_dir + "train-images-idx3-ubyte";
string label_path = base_dir + "train-labels-idx1-ubyte";

read_mnist_cv(img_path.c_str(), label_path.c_str());

Output as below: