Process objects asynchronously as they are returned from the database engine

Question

In EF 6, I would like to process objects asynchronously as they are returned from the database engine.

I know that I can name ToListAsync() and ForEachAsync() and maybe they do what I'm looking for, but I'm not sure. I think what I'm looking for is a combination of the two.

In my opinion, ToListAsync() will complete the task when the entire request is read from the database engine and converted to objects. This means that you need to wait for the entire request to return before the process begins.

I can’t determine if ForEachAsync() is doing what I am looking for, but what I assume due to the lack of information elsewhere is that ForEachAsync() just works with the set already received and processes each item is async.

What would be ideal is that ForEachAsync() (or another unknown method) will invoke the task as the data is retrieved from the database engine.

So ... is ForEachAsync() really doing this, and if not, is there any way to do this?

The reasons for this require two things:

• Large queries do not need to be stored in memory, because the whole collection, so saving on memory and the ability to process much larger results, not limited by memory
• The duration of the whole process is likely to be shorter due to the processing of each element during latency data retrieval

Update: Basically, if DbContext raised an event similar to OnEntityLoaded for each object when calling LoadAsync() , I could do whatever I wanted. Since I could just insert an object into a separate task processor, and entities could be processed efficiently and use any I / O delay. I can always set up a separate task processor, so I don’t need EF to support asynchronous processing of entities, just asynchronously load and fire an event / call a delegate when each object loads.

Update 2: And if ForEachAsync() was called when the entities loaded, it would also do what I need.

Answer 1

ForEachAsync , unlike ToListAsync , does not get all the elements in advance and just allows you to ToListAsync over it. Iteration of async itself.

QueryableExtensions.ForEachAsync delegates to IDbAsyncEnumerable.ForEachAsync which is in this :

 internal static async Task ForEachAsync( this IDbAsyncEnumerable source, Action<object> action, CancellationToken cancellationToken) { DebugCheck.NotNull(source); DebugCheck.NotNull(action); cancellationToken.ThrowIfCancellationRequested(); using (var enumerator = source.GetAsyncEnumerator()) { if (await enumerator.MoveNextAsync(cancellationToken).WithCurrentCulture()) { Task<bool> moveNextTask; do { cancellationToken.ThrowIfCancellationRequested(); var current = enumerator.Current; moveNextTask = enumerator.MoveNextAsync(cancellationToken); action(current); } while (await moveNextTask.WithCurrentCulture()); } } } 

You can see that this is very similar to how iterating over IEnumerable , but considering async-await . Instead of IEnumerable , GetEnumerator , IEnumerator and IDbAsyncEnumerable , we have IDbAsyncEnumerable , GetAsyncEnumerator , IDbAsyncEnumerator and IDbAsyncEnumerator .

MoveNextAsync allows you to actually retrieve items as needed as needed.