Mongodb response slows down incredibly after 200,000 entries

Question

At present, our task is to collect 1 million records from an external server, process it and save it in db. We use node.js to retrieve records and mongodb as a database.

We decided to divide the process into 2 tasks, extracting the records and processing them. Now we can extract all the records and upload them to the mongo, but when we try to process it (when processing, I mean changing several attribute values, simple calculations and updating attributes), we see a sharply slow response in mongodb being updated around 200,000 records.

To process the data, we take batches of 1000 records, process them, update the records (individually), and then go to the next batch. How can productivity be improved?

Answer 1

if you want to maintain response speed in mongoDB after long data then use mongo sharding and replication in your database and collection

Replication: - A replica set in MongoDB is a group of mongod processes that support the same dataset. Replica sets provide redundancy and high availability and are the foundation for all production deployments. This section introduces replication in MongoDB, as well as the components and architecture of replica sets. This section also contains tutorials for common tasks related to replica sets.

Replication link

Sharding: - Sharding is the process of storing data records on multiple computers and is MongoDBs' approach to meeting data growth needs. As the size of the data increases, one machine may not be sufficient to store data and provide acceptable bandwidth for reading and writing. Sharding solves the problem of horizontal scaling. With the box, you add more machines to support data growth and the requirements of read and write operations.

Sharding link