Reading a dataset file(Hex values) onto a block of memory-part 2

Question

I have a block of code that is trying to read the data from a dataset on to a randomly allocated block of memory. I don't know what exactly is inside the dataset but they access matrix values(Hex values) and put on to a memory location. And it works perfectly fine!

const unsigned int img_size = numel_img * sizeof(float);// (1248*960)4bytes= 4.79MB   
for (unsigned int i=0; i<p_rctheader->glb_numImg; ++i)// 0 to 496(Total no of images)
{
    const unsigned int cur_proj = i; // absolute projection number

    // read proj mx
    double* const pProjMx = pProjMatrixBuffers + cur_proj * 12;
    ifsData.read((char*) (pProjMx), 12 * sizeof(double));
    ifsData.seekg(img_size, ios::cur); 
        }

where pProjMatrixBuffers is

double** pProjMatrixBuffers = new double* [rctheader.glb_numImg];   //
pProjMatrixBuffers[0] = new double[rctheader.glb_numImg * 12];  //
for (unsigned int i=1; i<rctheader.glb_numImg; ++i) {
    pProjMatrixBuffers[i] = pProjMatrixBuffers[0] + i * 12;
}

There is a another read operation after this :

rctgdata.adv_pProjBuffers = new float* [num_proj_buffers];// 124 buffers
rctgdata.adv_pProjBuffers[0] = new float[num_proj_buffers * numel_img];// (1.198MB per image*124)*4bytes
    // set it to zero
memset(rctgdata.adv_pProjBuffers[0], 0, num_proj_buffers * numel_img * sizeof(float));
for (unsigned int i=1; i<num_proj_buffers; ++i) {
rctgdata.adv_pProjBuffers[i] = rctgdata.adv_pProjBuffers[0] + i * numel_img;
}

for (unsigned int i=0; i<numProjsInIteration; ++i)// (0 to 124)
{
const unsigned int cur_proj = numProcessedProjs + i; // absolute projection number// 0+124->124+124->248+124->372+124

// read proj mx
ifsData.read((char*) (pProjMatrixBuffers[cur_proj]), 12 * sizeof(double));
 // read image data
ifsData.read((char*) (rctgdata.adv_pProjBuffers[i]), numel_img * sizeof(float));
}

******EDITS**************************** Basically this code, reads Projection matrix from the dataset which is 12 doubles followed by 1248*960 image pixels.(floats). This goes on for 124 times inside for loop. Q1.If you see in the above code, pProjMatrixBuffers[cur_proj] is read twice, which could have been done once. (Correct me if I am wrong). Q2.How will rctgdata.adv_pProjBuffers[i] know where to start accessing the data from in the dataset? I mean location in the dataset. I am sorry if I have confused you. Please ask me for more information if needed. Thank you so much for all the help in advance!!

Answer 1

There is no way a 2-dimensional MxN-array can be allocated as such using new. The workaround in this code consists of an allocation of a 1-dimensional array of M pointers and another allocation of an array for the MxN elements. Then the M pointers are set to point to the M first elements of each row within the array for the elements.

Here we have two 2-dimensional arrays which I call (for obvious reasons) D and F. It's not clear how big D is - what's the value of rctheader.glb_numImg?

The first loop reads 12 doubles into a row of D and skips the float data for a row of F, doing a seekg with the appropriate positive offset to be added to the current position (i.e., forward). This is done rctheader.glb_numImg times.

There is something I don't see in this code: a single seekg back to the beginning of the file, after the first loop and before the second loop.

The second loop reads (once more) 12 doubles for each of the 124 rows and then, in one fell swoop, 1248*960 floats for each row. There is no need to reposition after these reads since the data for the second image immediately follows the data for the first image, and so on. (It's slightly irritating that num_proj_buffers and numProjsInIteration should have the same value, i.e., 124.)

It looks as if the second read loop would re-read what the first loop read. But since I don't know for sure that p_rctheader->glb_numImg is also 124, I can't really confirm that.

Calculating the size of what is read by 123 iterations of the second loop as

(1248*960*4 + 12*8)*124

this would account for ~0.5 GB - but the file size was reported as being ~2.5 GB.

Also note that one index within the second loop is computed as

unsigned int cur_proj = numProcessedProjs + i;

but the initial setting of numProcessedProjs is unclear.

Answer 2

To answer Q2, you allocate one big block of memory with new double[header.numImg * 12], and you also allocate a bunch of row pointers with new double* [header.numImg]. The first row pointer [0] points at the beginning of the memory (because it was used in the new call). The for loop then sets each row pointer [i] to point into the big block at 12-item increments (so each row should have 12 items in it). So for instance [1] points at the 12th item in the big block, [2] points at the 24th item, etc.

I haven't quite figured out what you mean by Q1 yet.