I can do this; I just don't know why it works. Using the MNIST database, which I downloaded from http://yann.lecun.com/exdb/mnist/, and the guidelines at the bottom of that page, I wrote the (as yet unfinished) method
// TRAINING SET IMAGE FILE (train-images-idx3-ubyte):
// [offset] [type] [value] [description]
// 0000 32 bit integer 0x00000803(2051) magic number
// 0004 32 bit integer 60000 number of images
// 0008 32 bit integer 28 number of rows
// 0012 32 bit integer 28 number of columns
// 0016 unsigned byte ?? pixel
// 0017 unsigned byte ?? pixel
// ........
// xxxx unsigned byte ?? pixel
// TEST SET IMAGE FILE (t10k-images-idx3-ubyte):
// [offset] [type] [value] [description]
// 0000 32 bit integer 0x00000803(2051) magic number
// 0004 32 bit integer 10000 number of images
// 0008 32 bit integer 28 number of rows
// 0012 32 bit integer 28 number of columns
// 0016 unsigned byte ?? pixel
// 0017 unsigned byte ?? pixel
// ........
// xxxx unsigned byte ?? pixel
let loadMnistImage file =
use stream = File.Open(file, FileMode.Open)
use reader = new BinaryReader(stream)
let magicNumber = readInt(reader)
let nImages = readInt(reader)
let nRows = readInt(reader)
let nColumns = readInt(reader)
(magicNumber, nImages, nRows, nColumns);;
That was the easy part. The difficult part is the form of the readInt
function. I can't just use BitConverter.ToInt()
; I found the answer in this page: https://code.google.com/p/aguaviva-libs/source/browse/c%23/NeuronalNetwork/sets/HandWriting.cs?spec=svn9ffdf444c6317be049572cea59170602c8f28bea&r=9ffdf444c6317be049572cea59170602c8f28bea.
Translating the method
int Read(BinaryReader b, int i)
{
int res = 0;
while (i-- > 0)
{
res <<= 8;
res |= b.ReadByte()
}
return res;
}
into F# gives
let readInt (b : BinaryReader) =
[1..4] |> List.fold (fun res item -> (res <<< 8) ||| (int)(b.ReadByte())) 0
(assuming i = 4
). This works: in F# interactive, the lines
loadMnistImage @"Data\t10k-images.idx3-ubyte"
loadMnistImage @"Data\train-images.idx3-ubyte"
give results of (2051, 10000, 28, 28)
and (2051, 60000, 28, 28)
respectively, which agree with the values in the comments from the first code snippet.
What I don't understand is why it works. What is with all this bit-shifting and folding on the bitwise or operator? Why can't I just use BitConverter.ToInt()
instead?