How to atomically increase a static member in IronPython?

Question

I have an IronPython script that uses TPL and Parallel.ForEach to process files using multiple threads. In C #, I can use Interlocked.Add and Interlocked.Increment to change global variables in an atomic thread-safe operation, but this does not work in IronPython because integers are immutable . I currently have a simple Results class that stores several variables as static members, which are used to track the results of a multithreaded operation. When changing multiple values, I can lock the class using the .NET Monitor class to ensure that the update is thread safe, but this seems like a lot of overhead if I only want to update one variable (for example, just increment the results. Files).

My question is, is there a better way to increment a single static member variable, like Results.Files in IronPython, in thread safe or atomic form, similar to how Interlocked.Increment works? Also, are there thread-safe counters built into the python or .NET framework that can be used in place of the base integer?

class Results: Files = 0 Lines = 0 Tolkens = 0 @staticmethod def Add(intFiles, intLines, intTolkens): #Use the System.Threading.Monitor class to ensure addition is thread safe Monitor.Enter(Results) Results.Files += intFiles Results.Lines += intLines Results.Tolkens += intTolkens Monitor.Exit(Results) #Finish thread safe code 

Answer 1

Well, it looks like python could use multiprocessing.Value , which by default would block the object on every access. Unfortunately, the multiprocessor class is not built into IronPython because it is based on CTypes. However, I found a way to do this using the Interlocked class and a reference to the CLR object:

 import clr from System.Threading import Interlocked refInt = clr.Reference<int>(5) #Create a reference to an integer #refInt = <System.Int32 object at 0x0000000000000049 [5]> #refInt.Value = 5 Interlocked.Increment(refInt) #Returns 6 and refInt now points to a new integer #refInt = <System.Int32 object at 0x000000000000004A [6]> #refInt.Value = 6 

In this case, you can use all the locking methods to add, compare, exchange, increase and read the refInt object. You can also get or set refInt.Value directly, but only blocked methods will be thread safe. In addition, blocked methods will NOT throw an overflow exception (it will simply be wrapped without pauses), so make sure you select a data type that is large enough to never overflow.

Answer 2

In the past, I divided work into parallel processing, saved the results with a unit of work, and compared them at the end. Think Map / Reduce, and you have it.

Create a new thread that takes your tuples as they arrive (or wait until everything is done). This incremental or complete summation method should be called at the end or be the only one that reads from the queue and increments counters.

Modify the add method to place the results as a tuple in the queue.

Hope this helps.

Jacob

Answer 3

If you want to use the C # bit, you can create a simple reusable C # class that encapsulates a member variable (non-static) int and provides lock functions.

 class InterlockedWrapper { private int _value; public int Increment() { return Interlocked.Increment(ref _value); } .... 

etc. Then you can use this class from Python.