An efficient way to store a large number of small objects

Question

I have a simple Person class with 4 lines and an integer.

 public class Person { public string FirstName { get; set; } public string LastName { get; set; } public int Age { get; set; } public string PostalCode { get; set; } } 

We need to store a large amount of data in memory. The collection must be searchable in any field. Items will be added and removed as part of the life cycle.

The Flyweight template does not work because the object does not have a ton of duplicate values, only at the field level. Which model or strategy will work best to limit memory overhead and work well?

Answer 1

We need to store a large amount of data in memory.

Then the Person[] array would be the fastest way, but the List<Person> would be close and much easier to work with. Just make sure you minimize redistribution using the Capacity parameter.

The collection must be searchable by any field.

Easy, .Where (p => p.FirstName == value) .
Speeding it up with dictionaries will cost memory.

Answer 2

A combination of things can work here ... First, it is best left as a reference type so that you do not copy structures during the search. Use string.Intern(string) to reduce memory usage by double names and postal codes ... Finally, use Dictionary<TKey, TValue> to index these entries by value ... Perhaps TKey is a string in The case of FirstName and TValue ' List<Person> , so you can search for people by the specified string ... This is called an inverted index: http://en.wikipedia.org/wiki/Inverted_index - an alternative to the dictionary - implement your own Tree structure or Trie, such as the trie prefix ... You trade O (log n) speeds for less memory than the O (1) dictionary.

From the point of view of storing LOT of them in memory, it depends on how many ... but in the end you want to have enough memory to process them ... Or start expanding access to distributed systems for sharing objects, for example, a template MapReduce or "paging" to disk.