Fast reading of an array of structs from a binary file

Question

Is it possible to read an array of structs from binary file in one call?

For example, I have a file containing thousands of vertices:

struct Vector3 { float x, y, z; }

I need C# port for the C++ code:

Vector3 *verts = new Vector3[num_verts];
fread ( verts, sizeof(Vector3), num_verts, f );

Answer 1

Here's one (of a few) ways:

void Main()
{
    var pts = 
        (from x in Enumerable.Range(0, 10)
        from y in Enumerable.Range(0, 10)
        from z in Enumerable.Range(0, 10)
        select new Vector3(){X = x, Y = y, Z = z}).ToArray();

    // write it out...
    var bigAssByteArray = new byte[Marshal.SizeOf(typeof(Vector3)) * pts.Length];
    var pinnedHandle = GCHandle.Alloc(pts, GCHandleType.Pinned);    
    Marshal.Copy(pinnedHandle.AddrOfPinnedObject(), bigAssByteArray, 0, bigAssByteArray.Length);
    pinnedHandle.Free();
    File.WriteAllBytes(@"c:\temp\vectors.out", bigAssByteArray);

    // ok, read it back...
    var readBytes = File.ReadAllBytes(@"c:\temp\vectors.out");
    var numVectors = readBytes.Length / Marshal.SizeOf(typeof(Vector3));
    var readVectors = new Vector3[numVectors];
    pinnedHandle = GCHandle.Alloc(readVectors, GCHandleType.Pinned);
    Marshal.Copy(readBytes, 0, pinnedHandle.AddrOfPinnedObject(), readBytes.Length);
    pinnedHandle.Free();

    var allEqual = 
        pts.Zip(readVectors, 
            (v1,v2) => (v1.X == v2.X) && (v1.Y == v2.Y) && (v1.Z == v2.Z))
        .All(p => p);
    Console.WriteLine("pts == readVectors? {0}", allEqual);
}


struct Vector3
{
    public float X;
    public float Y;
    public float Z;
}

Answer 2

Yes, it's possible, but you would have to add attributes to the structure so that you specify exactly how it's mapped in memory so that there is no padding in the struct.

Often it's easier to just convert the data yourself. The vast majority of the processing time will be reading the data from the file, so the overhead of converting the data is small. Example:

byte[] bytes = File.ReadAllBytes(fileName);
Vector3[] data = new Vector3[bytes.Length / 12];
for (var i = 0; i < data.Length; i++) {
  Vector3 item;
  item.x = BitConverter.ToSingle(bytes, i * 12);
  item.y = BitConverter.ToSingle(bytes, i * 12 + 4);
  item.z = BitConverter.ToSingle(bytes, i * 12 + 8);
  data[i] = item;
}