NextValue () performance counters Very slow (1000+ counters)

Question

In our application, we use Windows performance counters to store some of our application metrics, which are subsequently retrieved in some web services.

I have a problem with the amount of time it takes to read a value from counters. I looked through the rest of my application and everything is fine, the performance is wise, but reading from counters in a loop (from a list or array) takes a lot of time.

Code example:

// This triggers a read of the counter initial value (1000ms delay following for calculated counters) counters.ToList().ForEach(counter => counter.NextValue()); 

In my testing of only the cycle above, the list of 1,359 counters takes 20 seconds, and with a stopwatch in place, it seems that the average time to read the counter is 0-10 ms or about 80-90 ms. Many of them take 0 ms, the highest - about 170 ms, while the average non-zero is about 80-90 ms.

I may be too optimistic, but I would have thought that reading 1000 numerical values ​​should take only a few milliseconds. Is there more processing than I know?

In fact, I have another loop in my logic that gets the second value for the counters counted. It just makes it doubly worse. :)

Thanks!

---

Update 1

I wrapped the counter in the stopwatch and I am surprised by the results. Reading even a simple .RawValue property still takes too much time. I understand that counters basically work the same way, and the search should be incredibly fast; weird. I also see a template in which counters for Networking categories take longer.

According to http://joe.blog.freemansoft.com/2014/03/windows-performance-counters.html , performance for the performance counter service should also not be considered.

I posted some stopwatch results for the following pastebin: http://pastebin.com/raw.php?i=aDJk2Tru

My code is as follows:

 Stopwatch t; foreach (var c in counters) { t = Stopwatch.StartNew(); var r = c.RawValue; Debug.WriteLine(t.ElapsedMilliseconds.ToString("000") + " - " + c.CategoryName + ":" + c.CounterName + "(" + c.CounterType + ") = " + r); } 

As you can see in the paste, many readings are equal to 0, but in the range of 50-100 m. There are many. I really don’t understand how this can be. Of course, one counter value should be as fast as any other, right?

Answer 1

Here is what I managed to learn about the counters. Please forgive the grammar; this was somewhat extracted from the letter I sent on this issue.

• The processing time is 4-5 seconds on my machine at least (maybe better or worse on the server, not sure) to read instance names from the counter category. This varies slightly with the number of counters in the category. If you do not use instance counters, you can avoid this.
• We save all the counters in one category, so it is inevitable that this category will eventually have thousands of counters, given our situation. In my testing, the more counters in the category, the worse the performance. It seems like this should make sense, but the performance of a single counter is affected by the number of counters currently in memory, which is an odd correlation, maybe:
  • With 8 total counters, the reading time is about 1-2 ms per counter.
  • With 256 full counters, the read time is about 15-18 ms per counter.
  • With 512 total counters, the read time is about 30 ms per counter.
  • With 3,584 total counters (reading all counters), the read time is about 200 ms per counter.
  • With 3,584 total counters in the system (filtered in memory, with only 512 counters), the read time is from 50 to 90 ms per counter. Not sure why they are slower than the previous batch of 512 counters.
  • Several times I ran each of these tests using System.Diagnostics.Stopwatch for their time.
• It is important to note that the counters need to be read twice because many counters are calculated over a period of time and represent the average value between the start and end read times, so these bad numbers get worse in the real world scenario.

Given the numbers above, on my machine with 512 counters for about 50 ms each at the slower end, as well as the instance request and the second counter, we looked at about 60 seconds per request. This is given when working with only 512 counters at a time. Ive runs a full service request on my computer several times, and the request completes sequentially after 60-65 seconds.

Of course, I would not have expected such a decrease in the performance of individual counters based on the number of other counters that were evaluated. In my reading, the Windows Performance Monitor system should be fast, and with small collections it certainly is. Perhaps our precedent is not suitable, and we may abuse the system.

Update

Given that we have control over how we create counters, we decided to slightly change our approach. Instead of several categories with many counters, we instead create many categories, each of which has fewer counters (4-8 counters for each category). This approach allowed us to effectively avoid performance problems, and the counter reading time is in the range of 0-1ms. In our experience so far, even 100 new categories with several counters have not affected system performance in general.

It is important to note that when working with a large number of additional counters, you need to solve the memory limit set by default for performance counters. This can be done either through machine.config or in a registry entry. More information can be found here: http://msdn.microsoft.com/en-us/library/ms229387(v=vs.110).aspx