All geek questions in one place

Embarrassing parallelized tasks in .NET.

I am working on a problem when I need to perform many embarrassing parallelizable tasks. A task is created by reading data from a database, but the totality of all tasks will exceed the amount of memory on the machine, so tasks must be created, processed and deleted. I am wondering what would be a good approach to solve this problem? I think the following two approaches:

  • Deploy a synchronized task queue. Implement a manufacturer (task creator) that reads data from the database and puts the task in the queue (limit the number of tasks currently in the queue to a constant value to ensure that the memory size is not exceeded). Have several consumer processes (task processor) that read the task from the queue, process the task, save the result, and delete the task. What will be the large number of consumer processes in this approach?

  • Use the .NET Parallel extension (PLINQ or parallel for), but I understand that it is necessary to create a set of tasks (can we add tasks to the collection when processing in parallel for?). Thus, we will create a task package - say, N tasks at a time and process these batches of tasks and read N more tasks.

What are your thoughts on these two approaches?

+3
source share
6 answers

Use ThreadPool with a limited queue to avoid system overwhelming.

If each of your work tasks is associated with a CPU, then first configure your system so that the number of threads in your system is equal to the number of hardware threads that your box can execute.

If your tasks are not CPU related, you will have to experiment with the pool size to get the best solution for your specific situation.

You may need to experiment with any of these approaches to get to the optimal configuration.

Basically, check, tune, check, repeat until you are happy.

+4

PLINQ, , PLINQ (, vanilla LINQ) IEnumerable. , , , # (.. yield).

, , (, ), , PLINQ , . , , PLINQ ( " " ).

EDIT: PLINQ ThreadPool.

MSDN , ", TPL .

+3

ThreadPool.

, , . - .

+2

Microsoft HPC Server 2008. , , , - . HPC.

http://www.microsoft.com/hpc/en/us/default.aspx

+1

, .

? ?

, , , , , , ?

0

, Windows Workflow Foundation (WF) . , / .

-1

Source: https://habr.com/ru/post/1707867/

More articles:


All Articles