Mongoose (or similar ODM) in the registry of entries in memory?

Question

Specification

I have a MongoDB that contains a collection of records, call them operations for simplicity. Some of these operations running, and those that work contain a series of events that arrive in real time.

I emit these events in real time through socket.io , and also provide an API endpoint whose purpose is to provide an updated list of events.

Current situation

Watching how events arrive fast (up to a thousand per second) seems suboptimal for .save() recording (in this case, I use Mongoose as a mapper object) for each incoming event. The current situation is that I am compressing the .save() call only to execute execution every 2 seconds. Because of this, the list of requests by request is always somewhere between 0 and 2 seconds in the real-time stream when the operation continues.

Suggested Optimization

I am considering the possibility of creating a "registry" in the memory that contains links to all running operations (getting into the memory limits is hardly a concern, since there will be no more than 10 simultaneous operations in the foreseeable future).

Whenever a request arrives, the “registry” will first search for the entry, and if it is found, the latest version will be sent from there. If not, it will really query the DB.

---

TL; DR: the gap between real-time and on-demand events due to the throttle calls of model.save() , the proposed optimization is to use in-memory storage for a specific subset of records.

Question

Is this an effective optimization or will I skip the Mongoose point and possibly ignore other, more viable / relevant solutions?

Answer 1

Redis I would suggest taking a look at Redis if you haven't already taken this into account. You can use Redis to save records after they are saved in MongoDB and retrieved from Redis, if available, unless you request MongoDB and save redis again.

Or can Cassandra be used for both purposes?

Answer 2

As Vipin Dubi suggested, you can cache your queries using Redis . This is an in-memory data store that is commonly used for caching purposes.

I recommend the cachegoose module. It integrates the Mongoose caching concept perfectly, storing MongoDB queries (the ones you want) and retrieving them from the cache when you run the same query. This way you will have fewer calls in your database.

Redis also has various interfaces for visualizing your current values. I personally recommend Redsmin as it provides a cross-platform web interface with an awesome editor.