Mongodb poor performance

Question

I am currently using mongodb and I see very poor query performance (this may take a few seconds). The scenario is as follows:

I have structure documents:

{_id:"xxx", userId:"yyy", a:1 ,b:2, counter:1} 

In the test:

 "userId" value could be {1..200,000} "a" values could be {1..30} "b" values could be {1} 

Thus, my collection of maximum size will be 6,000,000. Currently, two indexes are defined for this collection: default _id and useId

Business logic queries all user records and then updates one specific one, incrementing the counter (query update is written "_id"). Also, if it is a new object, there is an insertion request.

I am working with mongo 1.8.2 on ubuntu with 8g ram

I have secondary secondary replicators (all mongo works with local disk storage and on the same subnet as the tomcat server). Of course, all readings pass to secondary ones and are written to the master. I did not test the fragments, because I think that 6,000,000 is not a huge collection, is it?

In addition, I run a jmetter test that generates 500 thread requests at a time with different userIds.

When I started mongostat, I see that% is locked very high (about 70%) after about 5-10 minutes of loading, I see that qw (write queue) is 500 (like the number of my open connections) When I stop the server, it takes mongo for about 10-20 minutes to complete all tasks in the queue.

I also ran db.serverStatus () and explained, and the results look fine. when I run db.currentOp (), I see requests that are waiting for a write lock. I could not get the currentOp output to parse it completely, because I ran the request from the command line and had only the window buffer size. But from there I saw a lot of updates (from _id) that are waiting for a lock record.

I would be grateful for any ideas.

One more thing: since each request will potentially bring 30 documents, I think there might be a different moddeling, as shown below:

 {_id:"xxx", userId:"123", bs: [{b:1, cs[{c:1, cnt:1}, {c:2, cnt:1}}, {{b:2 cs: [{c:1, cnt:1}]}}] 

But when I tried this simulation, I could not increase the counter, I just could not find the right way for this. I can do the insert, and the push of the bud cannot update for the following request:

 db.coll.update({userId:"123", "bs.b":1, "bs.cs.c":1}, {"bs.cs.cnt" : {$inc : 1}) 

I have an error about an illegal "point" in the request

I'm pretty complicated right now. Waiting for good ideas

Many thanks
Julia

Answer 1

MongoDB has a global write lock . This means that only one of your updates can continue at a time.

The db.serverStatus () command can help you diagnose problems with global write locks.

Here are some things you can try:

1) Make sure you are using mongodb 2.0. This is better concurrency than older versions. 2.2 would be better concurrency.

2) Leave the queue of your entries so that they are asynchronous and execute them all using a single thread. This can help with concurrency, because usually only one thread will try to use global write lock at a time.

3) If you are using the latest version and you cannot make your recordings single-threaded, then consider the shards. Sharding is more than just size; it is at least as important to write concurrency. If you outline, each segment will work in its own process with its own global write lock. This will allow the entire system to process more records.

Answer 2

To update, check out the positional operator :

 db.coll.update({userId:"123", "bs.b":1, "bs.cs.c":1}, {"bs.$.cs.$.cnt" : {$inc : 1}) 

To understand the cost of a request, use explain and make sure the requests are efficient.