Queue.Queue against semaphores, locks, etc. In Python multithreaded code

Question

The main task of multi-threaded applications is the coordination of threads that share data or other resources. To this end, a multi-threaded module provides a number of synchronization primitives, including locks, events, state variables, and semaphores.

Although these tools are powerful, minor design errors can lead to problems that are difficult to reproduce. So, the preferred approach to the coordination task is to concentrate all access to the resource in one stream, and then use the queue module to submit this stream with requests from other flows. Using Applications Queue.Queue objects for cross-thread communication and coordination are easier to design, more readable, and more reliable.

This basically indicates the use of Queue.Queue for inter-thread communication and coordination instead of powerful tools like semaphores, locks, etc.

My question is, what is the disadvantage of the proposed method? When to use more “powerful tools" and why?

Edit

To be clear, I know what semaphores are. I'm just wondering why the Python documentation suggests using the Queue.Queue method instead of the “powerful tools” - I just use my own documentation without coming up with my own.

Answer 1

I'm not sure I will consider semaphores and locks of the “more powerful methods" as you suggest.

Queues are usually a higher order abstraction. In other words, you can use semaphores and locks to create thread-safe queues.

What will you use when it depends on your application. Queues are good for transferring “work” between threads and processes, and semaphores / locks are good for protecting critical sections or shared resources, so only one thread can access at a time.

Answer 2

Check out the source code for the Python streaming queue . The queue class correctly creates an abstraction from 3 conditions and locks.

I would not say that coordination is the most difficult problem. In multi-threaded mode with a common state, the most difficult thing is preventing the "exchange" of threads. You should always monitor non-deterministic behavior due to random exchanges and stomp flows with each other.

So, I recommend you not to use streams at all. You should use lower-level tools if you think you have not spent enough time looking for heisenbugs, but if you can somehow avoid a simple queue, go for it.