How to create cross-process cross-processing in python?

Question

https://pypi.python.org/pypi/lockfile/0.12.2 :

This package has been deprecated . It is very preferable that instead of using this code base, which instead of fasteners _ or oslo.concurrency instead

However, fasteners understand that it is not thread safe:

( http://fasteners.readthedocs.org/en/latest/ )

Warning: there are no guarantees regarding the use of multiple threads in one process

And I can not find an example of using oslo.concurrency .

There is also speculation that using flock might resolve this situation, but the flock manual states:

( https://www.freebsd.org/cgi/man.cgi?query=flock&sektion=2 )

The flock system call applies or removes the advisory lock on the file associated with the fd file descriptor. A lock is applied by specifying an operation argument, which is one of LOCK_SH or LOCK_EX with the optional addition of LOCK_NB. To unlock an existing lock operation, there must be LOCK_UN.

Specific locks allow you to interact with processes to perform consistent file operations, but do not guarantee consistency (that is, processes can still access files without using advisory locks, which can lead to inconsistencies).

So...

Here's a python program that needs the lock and unlock functions, which will prevent the action from being executed by more than one thread in one process instance at a time.

(hint: starting with python test.py 1 & python test.py 2 & python test.py 3 )

How can I fix this code so that it works correctly?

 import sys import time import random import threading def lock(): pass # Something here? def unlock(): pass # Something here? def action(i): lock() id = threading.current_thread() pid = sys.argv[1] print("\n") for i in range(5): print("--> %s - %s - %s " % (i, id, pid)) unlock() class Worker(threading.Thread): def run(self): for i in range(10): action(i) for _ in range(2): Worker().start() 

The current, incorrect output is as follows:

 --> 0 - <Worker(Thread-2, started 123145310715904)> - 2 --> 3 - <Worker(Thread-1, started 123145306509312)> - 1 --> 0 - <Worker(Thread-2, started 123145310715904)> - 1 --> 1 - <Worker(Thread-2, started 123145310715904)> - 2 --> 2 - <Worker(Thread-2, started 123145310715904)> - 2 --> 1 - <Worker(Thread-2, started 123145310715904)> - 1 --> 4 - <Worker(Thread-1, started 123145306509312)> - 1 

and should look bigger:

 --> 0 - <Worker(Thread-2, started 123145310715904)> - 1 --> 1 - <Worker(Thread-2, started 123145310715904)> - 1 --> 2 - <Worker(Thread-2, started 123145310715904)> - 1 --> 3 - <Worker(Thread-2, started 123145310715904)> - 1 --> 4 - <Worker(Thread-2, started 123145310715904)> - 1 --> 0 - <Worker(Thread-2, started 123145310715904)> - 2 etc. 

Answer 1

Synchronization of related processes

If you can change your architecture to abandon your processes from the same parent, multiprocessing.Lock() should be enough. For example, this makes streams serial:

 lock = multiprocessing.Lock() def thread_proc(lock): with lock: for i in xrange(0, 10): print "IN THREAD", threading.current_thread() time.sleep(1) threads = [threading.Thread( target=functools.partial(thread_proc, lock)) for i in [1, 2] ] for thread in threads: thread.start() 

A potential problem may be that multiprocessing.Lock slightly underestimated. I cannot give you a specific link that multiprocessing.Lock objects are also suitable as thread blocking objects.

That said: On Windows, multiprocessing.Lock implemented using CreateSemaphore() , therefore, you get a cross-process lock. On Unix systems, you get a POSIX semaphore that has the same properties.

Portability can also be a problem, because not all * NIX systems have a POSIX semaphore (FreeBSD still has a port option for compiling Python without POSIX semaphore support).

See also Is there a reason to use threading.Lock over multiprocessing.Lock? and Martijn Pieters comment and answer in Why does python multiprocessing manager create thread locks?

Synchronize unrelated processes

However, as stated in your question, you have unrelated processes. In this case, you need a semaphore with the name, and Python does not provide these features (although it actually uses semaphores behind the scenes).

The posix_ipc library provides these features for you. Also seems to work on all relevant platforms.