Avoid the cumulative 16 MB limit

Question

I have a collection of about 1M documents. Each document has a property internalNumber, and I need to get everything internalNumberin my node.js code.

I used to

db.docs.distinct("internalNumber")

or

collection.distinct('internalNumber', {}, {},(err, result) => { /* ... */ })

in node.

But with the growth of the collection, I started getting an error: distinct is too big, 16m cap.

Now I want to use aggregation. It consumes a lot of memory and it is slow, but this is normal, since I need to do this only once when the script runs. I tried following in the Robo 3T GUI tool:

db.docs.aggregate([{$group: {_id: '$internalNumber'} }]); 

It works, and I wanted to use it in node.js code as follows:

collection.aggregate([{$group: {_id: '$internalNumber'} }],
  (err, docs) => { /* ... * });

But in the Node, I get an error: "MongoError: aggregation result exceeds maximum document size (16MB) at Function.MongoError.create".

Please help to overcome this limit.

Answer 1

, , , "" "", "".

"" .aggregate() BSON , :

let cursor = collection.aggregate(
  [{ "$group": { "_id": "$internalNumber" } }],
  { "cursor": { "batchSize": 500 } }
);

cursor.toArray((err,docs) => {
   // work with resuls
});

, .toArray(), JavaScript, "" "cursor" .

Answer 2

Casbah:

val pipeline = ...
collection.aggregate(pipeline, AggregationOptions(batchSize = 500, outputMode = AggregationOptions.CURSOR)