Best way to store lots of stock data in memory (variable)

Question

I'm trying to store a significant amount of stock market quote data into a variable for querying against in memory. I need to be able to quickly find out the most recent market data for a stock ID, and query specific lengths of history for a specific stock.

For example, I might receive data on stock ID 5342 (always numeric) every few seconds... my original thought is to build an array of SortedDictionary's, with the SortedDictionary key being the DateTime of the quote, and its value being my custom struct of market data. Then the outer array would be the stock IDs... so I could call:

RecentPrice = PriceData[StockID].Values.Last();

Or I could iterate backwards through that stock's SortedDictionary until I hit a key older than the time range I'm looking for.

However I feel like there has to be a better (more efficient) method. Any ideas?

Edit: Instead of an array of SortedDictionaries... a Dictionary of SortedDictionaries might be better. For example:

public static Dictionary<int, SortedDictionary<DateTime, StockData>> PriceData = 
    new Dictionary<int, SortedDictionary<DateTime, StockData>>();

then:

RecentPrice = PriceData[StockID].Values.Last();

Thanks!

Answer 1

Dictionaries, and hash tables in general, are good for exact matches. But when you want "the first date/time no earlier than X", a sorted list will perform best, because lookup is a binary search. Especially since you only are appending data, and not inserting it.

Answer 2

If your StockID values are contiguous and start at zero, array is likely to suffice. In the real-world, I imagine they're probably not, so a dictionary of dictionaries is good. I used them frequently for this sort of problem.

Answer 3

Have you thought about using a stack instead of a SortedDictionary? Some sort of custom implementation could work well if your data is always inserted in the correct order. Maybe a linked list.

If your data comes in sequentially why not simply store it in an array? That way you can use a binary search to quickly converge on your desired date range and your insert operation is very quick as well. It does waste a bit of memory though...

Answer 4

If you can ensure the new coming ticker data are in temporal order, SortedList is a better choice. It consume less memory and faster for insertion and removal ordered data.

Also if you need a variety of queries of the data. A in-memory database is a better choice. I use SqlLite to do the similar function in one of my projects and it handles different requirements very well cause I can use sql.