MathNet Numerics out of memory exception

Question

I'm using MathNet.Numerics in my F# project, and it runs out of memory when dealing with matrices that should be well within its remit.

EDIT: The problem is definitely not with MathNet.Numerics. It's something I'm doing, though I have yet to work out what that may be.

Here is how I'm building the MNIST matrix. Perhaps I'm leaving a file stream open or something, but I can't see how that would be the case.

let readInt (b : BinaryReader) =
    [1..4] |> List.fold (fun res item -> (res <<< 8) ||| (int)(b.ReadByte())) 0

let readImage (b : BinaryReader, rowArray, colArray) =
    rowArray |> List.collect (fun r -> List.map (fun c -> (b.ReadByte() |> int |> float)/255.0 ) colArray)

let loadMnistImage file =
    use stream = File.Open(file, FileMode.Open)
    use reader = new BinaryReader(stream)
    let magicNumber = readInt(reader)
    let nImages = readInt(reader)
    let nRows = readInt(reader)
    let nCols = readInt(reader)

    let row = [1..nRows]
    let col = [1..nCols]
    let images = [1..nImages] |> List.map (fun _ -> readImage(reader, row, col))
    matrix images;

Using this,

let mnistTrainingImages = loadMnistImage MnistTrainingImageData
let tr = mnistTrainingImages.Transpose()

blows up.

Is it them or is it me? I pulled the file from http://yann.lecun.com/exdb/mnist/, in case anyone fancies trying this themselves.

EDIT 1: The culprit is the call to loadMnistImage. According to Gene Belitski's suggestion, I ran the code

let rand = System.Random()
let m: double [,] = Array2D.init 60000 784 (fun _ _ -> rand.NextDouble())

before and after the line

let mnistTrainingImages = loadMnistImage MnistTrainingImageData

Before the line, it was fine; after the line, it blew up with the same out of memory exception. I would have thought that the use statements would release the file and I'd be OK (and this is not a massive file). However, I'm missing something, so I'll keep digging, and if anyone sees anything please do let me know.

EDIT 2: When I simplified the file reader as follows, the problem went away. It looks like the problem was the lazy evaluation of the list.

let readImage (b : BinaryReader) =
    (b.ReadByte() |> int |> float)/255.0

let loadMnistImage file =
    use stream = File.Open(file, FileMode.Open)
    use reader = new BinaryReader(stream)
    let magicNumber = readInt(reader)
    let nImages = readInt(reader)
    let nRows = readInt(reader)
    let nCols = readInt(reader)
    Array2D.init nImages (nRows * nCols) (fun _ _ -> readImage reader)

Answer 1

Even with 32-bit F#3.0 FSI the following snippet works for me without problems:

#if INTERACTIVE
#r @"....\packages\MathNet.Numerics.2.6.2\lib\net40\MathNet.Numerics.dll"
#endif

open MathNet.Numerics.LinearAlgebra.Double

let rand = System.Random()
let m: double [,] = Array2D.init 60000 784 (fun _ _ -> rand.NextDouble())
let mx = DenseMatrix.OfArray m
let tmx = mx.Transpose()

Must work in your setup, does it?