Batch Processing on MVC5 Website

Question

I have an MVC website using EF to access data. The application receives data, starts a series of calculations and saves the results. Each batch of data can have several thousand records, and calculations - on average 30 seconds - I want to run all this in the background.

For now, I have Hangfire to run games. Then I do:

var queue = new Queue<MyItem>();

// queue is populated ...

while (queue.Any())
{
    var item = queue.Dequeue();
    var task = Task.Run(() =>
    {
        using (var context = new MyDbContext())
        {
            context.MyItem.Add(item);

            // Run Calculations

           try {
               context.SaveChanges();
           }
           catch {
               // Log error
           }
        }
    }
}

When the package is launched, the site either completely stops responding, or I get the message "Base provider error with Open errors."

Is there a better approach to this?

Answer 1

, , Task.Run . , , ThreadPool. , (, , ) .

- concurrency . TPL ActionBlock. (, MaxDegreeOfParallelism), :

block = new ActionBlock<MyItem>(item =>
{
    using (var context = new MyDbContext())
    {
        context.MyItem.Add(item);

        // Run Calculations

       try {
           context.SaveChanges();
       }
       catch {
           // Log error
       }
    } 
}, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 2 });

while (queue.Any())
{
    var item = queue.Dequeue();
    block.Post(item);
}

block.Complete();
await block.Completion;