Exchanging big lists between C# and F#

Question

I have big data files of historical stock data, that I want to load over a C# GUI and then process with different algorithms in F#. Additional values are appended to lists over time.

The lists are two-dimensional and in the form

[[id, Open,High,Low,Close], [id, Open,High,Low,Close], ...]

The F# code is in a library and cannot work with C# lists (I cannot get the values of the second dimension of the list in F#). Converting the entire list every time a F# operation takes place, is too expensive. The values cannot be stored in a F# list either, because C# can't handle them.

Proposed solutions are:

1. either storing an F# list in C#, just for storage purposes since the library can't store values, together with a C# list containing the same values, converting parts of the list as needed. The main problem here is that the files can be multiple GB in size.

2. reading the values that are needed directly from a file in F# and saving calculated values in additional files. The problem here is the efficient organisation of the files, so that a selection of lines (e.g. for a moving average calculation) to load can be done quickly.

Answer 1

Does the inner sequence need to be a list? If it always has the same elements (OHLC) it would be better to use a tuple (or a custom type) since it makes explicit its fixed nature. I would probably do something like this:

module Stock =
  let (!) s = DateTime.Parse(s)

  let getOHLC() : unit -> seq<DateTime * float * float * float * float> =
    seq [
      !"18-Dec-12",27.25,27.63,27.14,27.56
      !"17-Dec-12",26.79,27.22,26.68,27.10
      !"14-Dec-12",27.11,27.13,26.70,26.81
      //...
    ]

If you really need each element to be a list, you can pipe it through Seq.map seq.

The seq function is defined as:

let seq (x:seq<'T>) = x

so it only upcasts, not copy.

Answer 2

Converting the entire list every time a F# operation takes place, is too expensive.

Have you measured it?

The values cannot be stored in a F# list either, because C# can't handle them.

You should be able to enumerate over the elements of an F# list from C# because it is just another enumerable.

The type in C# is System.Generics.List<System.Generics.List<Double>> and is passed as IEnumerable to F#. Isn't calling OfSeq() converting the list value, forcing me to loop through the entire list calling OfSeq() on every element and thus duplicating it.

Calling something like Array.ofSeq would create a new collection but you don't need to do that to iterate over the contents.

Note that the F# name for the type you gave is ResizeArray<ResizeArray<float>>.

For example, you can do:

let f (xss: ResizeArray<ResizeArray<float>>) =
  for xs in xss do
    for x in xs do
      printfn "%A" x

to consume your data from F# without copying.