Best way to store large amounts of stock data in memory (variable)

Question

I am trying to store a significant portion of stock market quotation data in a variable for a query in memory. I need to quickly find out the latest market data for a stock identifier and request specific history lengths for a specific stock.

For example, I could get data on a stock ID 5342 (always numeric) every few seconds ... my original thought was to build an array of SortedDictionary, with the SortedDictionary key being a DateTime quote, and its value is my custom market data structure. Then the external array will be the stock identifier ... so that I could call:

RecentPrice = PriceData[StockID].Values.Last(); 

Or I could scroll back through this stock of SortedDictionary until I press a key older than the range of time I'm looking for.

However, I feel that there should be a better (more efficient) method. Any ideas?

Edit: Instead of an array of SortedDictionaries ... A SortedDictionaries dictionary could be better. For instance:

 public static Dictionary<int, SortedDictionary<DateTime, StockData>> PriceData = new Dictionary<int, SortedDictionary<DateTime, StockData>>(); 

then

 RecentPrice = PriceData[StockID].Values.Last(); 

Thanks!

Answer 1

Dictionaries and hash tables are generally good for exact matches. But when you want “first date / time not earlier than X”, a sorted list will work best, because a search is a binary search. Moreover, you only add data, but do not insert it.

Answer 2

If your StockID values ​​are contiguous and start from scratch, an array may be enough. In the real world, I think they are probably not, so the dictionary of dictionaries is good. I often used them for this problem.

Answer 3

Have you thought about using a stack instead of a SortedDictionary? Some special implementation may work well if your data is always inserted in the correct order. Perhaps a linked list.

If your data comes in sequentially and not just store it in an array? Thus, you can use binary search to quickly converge to the desired date range, and your insert operation is very fast. This is a little scary memory, though ...

Answer 4

If you can guarantee that new quote data is temporary, SortedList is the best choice. It consumes less memory and is faster for inserting and deleting ordered data.

Also if you need various data requests. A memory database is the best choice. I use SqlLite to perform a similar function in one of my projects and handles various requirements very well, because I can use sql.