Is python a serious option for concurrent programming

Question

just thinking about starting to learn python, but I have one problem before I invest more time. Let me talk about this as a statement, followed by concern for others to comment, as the assumptions in the statement may be invalid:

I read about GIL, and there seems to be consensus, if you need concurrent solutions in python, it is best to fork a new process to avoid GIL.

I am worried that if I have a problem, I would like to divide into N * 2 pieces by N processors (suppose, for example, I have one server that runs * nix o / s with 8 cores). the context of switching fines between processes rather than between threads, which is more expensive, which limits performance.

I ask about this because there are other languages ​​out there that claim to be superior in such a scenario, and I wonder if this is suitable for this arena.

Answer 1

Python is not very good for parallel programming related to the processor. GIL (in many cases) makes your program work as if it worked on a single core - or even worse. Even Unladen Swallow (maybe) will not solve this problem (quote from our project plan : “We are no longer so optimistic about our chances of removing GIL completely”).

As you already said, other languages ​​claim to work better in parallel programming. Haskell , for example, has built-in functions for programming parallel applications. You can also try C ++ with OpenMP, which I think makes parallelization very easy.

If your application is connected to I / O, Python can be a serious decision, since GIL is usually freed up when blocking calls.

Answer 2

multiprocessing can get around the GIL, but it introduces its own problems, such as communication between processes.

Answer 3

In my limited experience, the “context switch cost” is reassessed as a performance limitation.

I / O bandwidth and memory are the most common limiting factors. Python I / O is comparable to many other languages, since it just uses the standard C libraries quite simply.

Your actual problem may not be typical. However, many problems work very well in multiprocessing mode, since they are actually related to I / O. Often this is the file system, reading a web page, or database operations that limit performance long before the context switches.

Answer 4

If you plan to learn Python to solve this problem, I could suggest a look at Erlang. It perfectly supports very lightweight processes and built-in primitives for IPC.

Do not let you learn Python, of course, simply by suggesting what might be the best tool for this particular task.

Answer 5

Also, if you are looking at sharing objects between python processes, I suggest you look at Alex's answer to this question

Answer 6

writing codes for several processes is not an easy task.

But if you start thinking like that, first, in the end, it’s easier to scale if one machine is not enough ... Threads cannot be used on different machines ...

Answer 7

Very quick one-word answer: Golan.