Migrating from EDU to java.util.concurrent reduces performance twice

Question

Cross mail from http://forums.oracle.com/forums/thread.jspa?threadID=2195025&tstart=0

There is a communications application server (JAIN SLEE) and an application running on it.
The application receives a message from the network, processes it, and sends a response back to the network.
The requirement for request / response delay is 250 ms for 95% of calls and 3,000 ms for 99.999% of calls.
We use EDU.oswego.cs.dl.util.concurrent.ConcurrentHashMap, 1 instance. The following methods are processed for one call (one call - several messages):

"put", "get", "get", "get", then in 180 seconds "remove". 

There are 4 threads that call these methods.
(A short note: working with ConcurrentHashMap is not the only action. There are also many other actions for one network message: parsing protocol messages, querying the database, writing SDR to a file, creating short live and long live objects.)

When we go from EDU.oswego.cs.dl.util.concurrent.ConcurrentHashMap to java.util.concurrent.ConcurrentHashMap, we see a performance drop from 1400 to 800 calls per second.
The first bottleneck for the last 800 calls per second is insufficient latency for the above requirement.

This performance degradation is reproduced on hosts with the following processor:

• 2 CPU x Quad-Core AMD Opteron 2356 2312 MHz, a total of 8 HW streams,
  
• 2 Processor x Intel Xeon E5410 2.33 GHz, 8 HW overall.
  

It does not play on the X5570 processor (Intel Xeon Nehalem X5570 2.93 GHz, 16 HW streams in total).

Has anyone encountered similar problems? How to solve them?

Answer 1

I assume that you take nanoseconds, not milliseconds. (This is a million times smaller!)

OR using ConcurrentHashMap is a trivial part of your delay.

EDIT: edited the example to be multithreaded using 100 tasks.

 /* Average operation time for a map of 10,000,000 was 48 ns Average operation time for a map of 5,000,000 was 51 ns Average operation time for a map of 2,500,000 was 48 ns Average operation time for a map of 1,250,000 was 46 ns Average operation time for a map of 625,000 was 45 ns Average operation time for a map of 312,500 was 44 ns Average operation time for a map of 156,200 was 38 ns Average operation time for a map of 78,100 was 34 ns Average operation time for a map of 39,000 was 35 ns Average operation time for a map of 19,500 was 37 ns */ public static void main(String... args) { ExecutorService es = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); try { for (int size = 100000; size >= 100; size /= 2) test(es, size); } finally { es.shutdown(); } } private static void test(ExecutorService es, final int size) { int tasks = 100; final ConcurrentHashMap<Integer, String> map = new ConcurrentHashMap<Integer, String>(tasks*size); List<Future> futures = new ArrayList<Future>(); long start = System.nanoTime(); for (int j = 0; j < tasks; j++) { final int offset = j * size; futures.add(es.submit(new Runnable() { public void run() { for (int i = 0; i < size; i++) map.put(offset + i, "" + i); int total = 0; for (int j = 0; j < 10; j++) for (int i = 0; i < size; i++) total += map.get(offset + i).length(); for (int i = 0; i < size; i++) map.remove(offset + i); } })); } try { for (Future future : futures) future.get(); } catch (Exception e) { throw new AssertionError(e); } long time = System.nanoTime() - start; System.out.printf("Average operation time for a map of %,d was %,d ns%n", size * tasks, time / tasks / 12 / size); } 

Answer 2

At first you noticed that the hash map is really the culprit? Assuming you did this: there is a hash map lock designed to scale up to hundreds of processors without introducing much controversy. He is the author of Cliff Click on a renowned engineer in the original Hot Spot compiler team. Now, working on JDK scaling on machines with hundreds of processors. Therefore, I assume that he knows what he is doing in the implementation of the hash map. More information about this hash map can be found in these slides .

Answer 3

Have you tried changing th concurrencyLevel in ConcurrentHashMap? Try a few lower values, like 8, try some larger values. And remember that performance and concurrency ConcurrentHashMap depend on the quality of the HashCode function .

And yes, this is java.util.ConcurrentHashMap has the same origin (Doug Lee from edu.oswego) as edu.oswego.cs.dl ... but it has been completely rewritten by him so that it can scale better.

I think it might be useful for you to check the fast-forward card. This may be better for real-time applications.