A memory leak in a large array. Will the IList subclass be fixed?

Question

I need to improve the memory performance in my application, and I could see that I have problems with memory fragmentation.

I read an interesting article about large objects from Andrew Hunter of Red Gate, and one of the solutions he recommends:

If you need large data structures for a long time, and especially if they need to grow over time, the best approach is to simply consider using or writing a data structure to store them. Arrays can contain up to 10,000 elements before they are a large heap of an object and can cause problems, so a very efficient way to store 100,000 records can be to store 10 arrays, each of which contains 10,000 elements: none of them will be a big heap object, so no fragmentation will happen. This can be written like this: a subclass of IList that will make it easy to carry transparently replace existing code.

How to implement his sentence in my code?

My program has a very complex form (with an object that leaves a residual memory every time it opens). I found a complicated list that might be the culprit, and I would like to implement his suggestion to find out if it fixes the problem.

Answer 1

What is wrong with using List for this? This is nothing but an implementation of IList, and you can do the splitting yourself. But if you want to do it transparently:

Implement IList (it's just an interface, nothing special about that. Maybe I don’t understand the question?) And reinforce it with arrays of the desired size. Then your Get() will take the index / sizeOfArrays index as the index of the array containing the element you index % sizeOfArrays , and return the index % sizeOfArrays th element in that array.

---

For pleasure, because it's lazy Friday, I wrote something. Note:

• I have not tested it
• I can’t comment on the correctness of your statements that this can help avoid memory fragmentation, I just blindly looked at your request.
• I don't know if List or any other collection is already smart enough to do just that.
• I made some decisions that may be wrong for you (that is, you cannot blindly refuse this in your code if you are now using arrays. Look at the implementation of Item , especially the installer, for example

However, here is the starting point, which reduced my motivational deficit before leaving. I left some interesting methods as an exercise for a respected reader (or OP) ..; -)

 public class PartitionList<T> : IList<T> { private readonly int _maxCountPerList; private readonly IList<IList<T>> _lists; public PartitionList(int maxCountPerList) { _maxCountPerList = maxCountPerList; _lists = new List<IList<T>> { new List<T>() }; } public IEnumerator<T> GetEnumerator() { return _lists.SelectMany(list => list).GetEnumerator(); } IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); } public void Add(T item) { var lastList = _lists[_lists.Count - 1]; if (lastList.Count == _maxCountPerList) { lastList = new List<T>(); _lists.Add(lastList); } lastList.Add(item); } public void Clear() { while (_lists.Count > 1) _lists.RemoveAt(1); _lists[0].Clear(); } public bool Contains(T item) { return _lists.Any(sublist => sublist.Contains(item)); } public void CopyTo(T[] array, int arrayIndex) { // Homework throw new NotImplementedException(); } public bool Remove(T item) { // Evil, Linq with sideeffects return _lists.Any(sublist => sublist.Remove(item)); } public int Count { get { return _lists.Sum(subList => subList.Count); } } public bool IsReadOnly { get { return false; } } public int IndexOf(T item) { int index = _lists.Select((subList, i) => subList.IndexOf(item) * i).Max(); return (index > -1) ? index : -1; } public void Insert(int index, T item) { // Homework throw new NotImplementedException(); } public void RemoveAt(int index) { // Homework throw new NotImplementedException(); } public T this[int index] { get { if (index >= _lists.Sum(subList => subList.Count)) { throw new IndexOutOfRangeException(); } var list = _lists[index / _maxCountPerList]; return list[index % _maxCountPerList]; } set { if (index >= _lists.Sum(subList => subList.Count)) { throw new IndexOutOfRangeException(); } var list = _lists[index / _maxCountPerList]; list[index % _maxCountPerList] = value; } } }