Creating a large number of Sitecore items causes a memory leak

Question

I have a Sitecore application where I need to create a large number of Sitecore elements in a loop from another system. Below is my test assignment cycle;

using (new Sitecore.SecurityModel.SecurityDisabler()) { for (int i = 0; i < 20000; i++) { Item newItem = MyTemplateItem.CreateItemFrom(DateTime.Now.Ticks.ToString(), myParentItem); newItem.Editing.BeginEdit(); newItem.Fields["Title"].Value = Guid.NewGuid().ToString(); newItem.Editing.EndEdit(); } } 

When this cycle is started and when viewed in the task manager, the memory usage of the process increases over time.

The Sitecore Item class did not implement the IDisposable interface, so I cannot call the Finalizer of the created item.

How can I avoid a memory leak?

PS: I am using a Windows application to perform this operation to bypass the memory leak of the IIS process, where Sitecore is updating the cache and building the index, which should be done at the end of this process.

Answer 1

You can temporarily disable the database cache when creating an item:

 using (new Sitecore.Data.DatabaseCacheDisabler()) 

You can disable indexing at creation time using:

 Sitecore.Configuration.Settings.Indexing.Enabled = false; 

Try adding these two things and see if it helps.

UPDATE: Maybe this will work. Add this inside the loop after EndEdit ():

 newItem = null; if (i % 100 == 0) { GC.Collect(); GC.WaitForPendingFinalizers(); } 

This will force the garbage collector to try to recover unused memory each time 100 items are added.

If this works, you really do not need to call GC yourself, it will happen automatically at some point, you just don't know when.

Answer 2

Ruud's answer seems to work better than it was. Memory growth was fast due to the creation of a database cache. Disabling sharply slows down memory growth, but nonetheless, growth is very slow.

I used the following statements for this:

 using (new Sitecore.Data.DatabaseCacheDisabler()) newItem = null; // did not worked // replaced above line with the below Marshal.Release(Marshal.GetIUnknownForObject(newItem)); if (i % 100 == 0) { GC.Collect(); GC.WaitForPendingFinalizers(); } 

Thanks to all the supporters!

Answer 3

If you set your cache limits to high (i.e., too high) and do what you do, it would be wise to get the symptoms you report. If you look at the /sitecore/admin/cache.aspx page while your script is running, this will be shown. I would like the master[items] and master[data] records to keep rising ...

If this case, just setting a lower cache limit would mean that the caches are lowered as needed, which seems like a healthier way to keep things under control.

In addition, if you want to significantly speed up the creation of program elements, try using

using (new BulkUpdateContext()) { code.. }

BulkUpdateContext () may also skip adding items to the cache - I have not checked if this is the case, but if so, it will also most likely fix the memory problem.

(also see this SO answer ). Note that it disables multiple pipelines.