Can a limited lock display data during upload

Question

I have a BlockingCollection (ConcurrentBag, 50000) where I am trying to use a very small limited capacity of 50,000 for producer threads in order to maximize the number of entries that I can process in my ConcurrentDictionary consumer stream. A producer is much, much faster than a consumer, and otherwise consumes most of the memory.

Unfortunately, I immediately noticed that the total number of entries in my ConcurrentDictionary is now significantly lower than it should be after adding a limited capacity of 50,000 when running my test data. I read that the BlockingCollection.add method should block indefinitely until there is room for the collection to add. However, this does not seem to be the case.

Questions:

• Will the BlockingCollection.add method eventually fail or silently work if too many add is called before the capacity in the BlockingCollection is empty?

• If the answer to # 1 is yes, how many add-ons can I try after exceeding the bandwidth without losing data?

• If many BlockingCollection.add () methods are called that expect / block capacity and the CompleteAdding () method is called, will the wait / lock wait continue to wait, and then, in the end, add or make them fail?

Answer 1

Make sure that if you use BlockingCollection together with ConcurrentDictionary, you do not have the BlockingCollection.TryAdd (myobject) method, which is hidden in your code somewhere and is mistaken for the ConcurrentDictionary.TryAdd () method. BlockingCollection.TryAdd (myobject) will return false and cancel the add request, creating a silent failure if the BlockingCollection bandwidth is exceeded.

• The BlockingCollection.Add () method does not appear to “delay too much” or lose adding after exceeding the Limit capacity by a large amount. The add () method will eventually cause the process to end due to out of memory if too many .add () are waiting to be added to the BlockingCollection, which exceeds the capacity. (This should be a very extreme case of flow control problems).
• See No. 1.
• My own tests show that after calling CompleteAdding (), all subsequent additions fail, as described in the MSDN docs.

Final Note on Performance

It seems that (in my own case anyway) using Bounding Capacity and .Add () on the BlockingCollection is very slow compared to using Bounding Capacity and .TryAdd () in the same process.

I achieved much better results by implementing my own bandwidth limitation strategy. There are many ways to do this. Three options include Thread.Sleep (), Thread.Spinwait (), or Monitor.Wait (), used with Monitor.PulseAll (). When using one of these strategies, you can also use BlockingCollection.TryAdd () instead of BlockingCollection.Add () and not have bandwidth without data loss or out of memory. This method also seems to provide better performance.

You can choose one of three examples based on which scenario is best suited for the speed differences in your Producer and Consumer streams.

Thread.Wait () Example:

//Check to see if the BlockingCollection bounded capacity has been exceeded. while (Tokens.Count > 50000) { //If the bounded capacity has been exceeded //place the thread in wait mode Thread.Sleep(SleepTime); } 

Thread.SpinWait () Example:

 //Check to see if the BlockingCollection bounded capacity has been exceeded. while (Tokens.Count > 50000) { //If the capacity has been exceeded //place the thread in wait mode Thread.SpinWait(SpinCount); } 

Monitor.Wait () example

In this example, a hook is required on both the producer and consumer side.

Manufacturer Code

 //Check to see BlockingCollection capacity has been exceeded. if (Tokens.Count > 50000) { lock (syncLock) { //Double check before waiting if (Tokens.Count > 50000) { Monitor.Wait(syncLock, 1000); } } } 

User code

 //Check to see BlockingCollection capacity is back a normal range. if (Tokens.Count <= 40000) { lock (syncLock) { //Double check before waiting if (Tokens.Count < 40000) { Monitor.PulseAll(syncLock); } } }