How should I implement “get objects changed from the moment” using MongoDB?

Question

I have a set of objects, let's say these are “messages”, and these objects can be changed. I would like to display a list on the client side that is being updated dynamically. Thus, on the client side, if you do this through a poll, the client will call the API as:

getPostsChangedSince(serial) 

where serial can be a monotonously increasing number, probably a timestamp. The client returns a list of messages that have changed since that time, stores the new last serial number, and the next time the client polls, it requests changes from the moment of the last serial number.

I think the main idea in this question is the same (we are talking about ASP.NET): How to implement "get lattests with changed elements" with ADO.NET data services?

I am trying to find a better way to implement this in MongoDB.

I like the idea of ​​using time for the series, because it automatically works, at least mostly correctly, even if there are several application servers. The serial number will be stored in each post-object and updated with every change in the object.

Timestamp based serial number can be implemented as:

• a date (I think it is stored as a 64-bit millisecond from an era?)
• a Timestamp http://www.mongodb.org/display/DOCS/Timestamp+Data+Type
• something "manually", for example. store milliseconds as a number

Some useful features to solve will include:

• make sure that creating and immediately updating an object in OS timer resolution will continue to increase the number of serial numbers, even though it is equal to one time
• it would be even better to guarantee a monotonic increase on a global scale for all objects, and not only guarantee that changing this object will lead to a failure on this object (in the absence of calls to getPostsChangedSince (), it will probably take a while to avoid omitting the change - by the price of receiving some changes twice)
• Mongodb timestamps can be enjoyable because getting time in the application creates a gap between when you get time and when a new object is saved and available in requests
• using findAndModify () with the query, including the old serial one, so "conflicts" (two changes at once) will cause an error that allows the application to retry

I understand that some of the corner cases here are a bit “academic” and probably can be tricked into real life.

So far my approach is:

• use date type for sequential
• when changing the object, get the current time, and if it matches the object of the old serial number, add 1 millisecond (yes, it breaks if you make two modifications quickly without reinstalling from mongodb, but it seems OK)
• use findAndModify (), but based on https://jira.mongodb.org/browse/JAVA-276 there can be no way to determine if it will end with something, change (i.e. the second change is ignored, in case of conflict )

Questions:

• I feel that I should use Timestamp instead; true? Any flaws?
• If you have a mongo cluster, can the time in milliseconds be more unique and correct than the Timestamp time in seconds plus a number, and with one mongod Timestamp is more unique?
• Is there a way to find that findAndModify () has updated something?
• Any general tips / experience with this problem? how would you do that?

Answer 1

Have you considered the “externalization” of the serial number generator? Time with MongoDB precision is good, but it can be difficult to synchronize when using multiple machines. One option is that you can use memcached or something similar based on memory, very fast and can be serialized (memcached has a CAS operation).

So, what would you do is save the "seed" in memcached using the say, counter key. Each time the application needs to do an insert, it gets the next number from memcached and increments the counter.

Secondly, you can even do away with memcached and just use a single line collection (sorry document) that has a counter. You can get the counter and increase it, which will be an extremely fast operation simulating memcached.

And, of course, you can index the data accordingly. However, I'm curious that this will cause the index to be very unbalanced (right side babble). Depending on the situation, it might be worth considering a limited fee. Therefore, when you insert data into your main collection, also insert it into a limited collection and read the data from this collection.

Answer 2

You can continue to use your regular collection, as it is now, and after each update, additionally insert a message identifier in a special TTL collection. See http://docs.mongodb.org/manual/tutorial/expire-data/ for more information on using such a collection. Mongo will take care of all synchronization issues, you don’t need to worry about serial numbers, and you can very quickly access lists of objects by time by their identifiers.

Protest:

use the findAndModify blocking form to verify that the changes are actually processed:

Lock / Secure Records

If you do not specify the “new” parameter as true, the write operation will not be blocked and will not return an error (if any). If you want the “new” document to be returned, the operation will wait until the record is completed to return a new document or error.

For a "safe" (blocking) write operation, you must call getLastError (unless you use the "new").