Boost :: thread_resource_error with more than 32705 threads

Question

I am implementing a message passing algorithm. Messages are distributed through the nodes of the chart, blocking until they have received sufficient information (from other neighbors) to send the message.

It is easy to write the algorithm if I put each message in my stream and use the boost :: condition to pause the stream until all the necessary information is available. I create many thousands of threads, but in most cases, only some of them are active at any time. This seems to work very well.

My problem is that during unit testing, I found that if I create more than 32705 threads, I get

unknown location (0): fatal error in "Tree_test": std :: exception: raising :: thread_resource_error

, and I don’t know what causes this, or how to fix it.

It seems that there is available memory (each thread contains only two pointers - the objects through which the message passes).

From this question: What is the maximum number of threads for a process on Linux? I think the following information is relevant (although I really don't know what that means ...)

~> cat /proc/sys/kernel/threads-max 1000000 

(I increased this from 60120 - do I need to restart?)

  ~>ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 

I tried to bother with waiting signals (my limit is very close to 2 * that number) and the stack size with ulimit -S -i 8191 - (I could not increase it), but these changes did not seem to affect at all)

I'm on 64 bit Ubuntu-10-10 if that helps ...

Answer 1

I think that with 32K threads in the system you should look for potential solutions other than having more threads. For example, a thread pool (Boost has some things for this).

Anyway, in your system there are no PIDs limited to 32768 or some such value? I hope that you will finish sooner or later, may also develop a system that can handle more elements than the maximum number of threads.

So, look at / proc / sys / kernel / pid _max to see your maximum PID, and try increasing it. This can lead to over 32K (but it can also cause unexpected behavior with programs not designed for unusually large PIDs, so be careful).

And then you can be limited by stack space (as opposed to virtual memory space). You can try creating streams with smaller stacks if you want.

Answer 2

Good to answer the question: do you need to increase

 /proc/sys/vm/max_map_count 

As discussed here:

https://listman.redhat.com/archives/phil-list/2003-August/msg00025.html

and here:

http://www.kegel.com/c10k.html#limits.threads

HOWEVER: FOR THE BEST WAYS TO DO THIS, SEE THE FOLLOWING QUESTION:

Non-threaded alternative to anticipation provided. (Edit: proactor template with boost.asio?)

Answer 3

Actually, it depends on how big your stacks are, but when you create a large number of threads, you will run out of address (32-bit) or virtual memory (64-bit).

On Linux pthreads, the default stack size was 10Mb the last time I checked; this means that 32k streams use 320G of address space (note that it will probably be lazily initialized, so it won’t use that much virtual memory); it's probably too much.

Even if you make the stack pretty small and don't run out of memory this way, 32k threads will use a lot of virtual memory for the stacks. Consider using a different approach.

ulimit only affects the stack size of the source stream (which is usually dynamic under Linux); the stack size of other threads is fixed and set at the time the pthread library created the thread.