Python Mongo "Sort operation is used more than maximum" when skipping is high

Question

I have code that makes a fairly simple request-skip-limit-sort. I encounter phenomena that are difficult for me to explain.

In the "small" value of the pass - everything is in order. In the "high" skip value (> 18000) - I can not get a result with a limit above 20 without receiving the following error:

OperationFailure: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.

The question is, why does this happen only with a large count of passes? How can i solve this?

Works on mongoShell (even with DBQuery.shellBatchSize = 300). And it looks like it uses the db.my_collection.find index ({'foo': false}). skip (19000) .limit (100) .sort ({'meta_data.created_at': - 1}). explain ()

"queryPlanner" : {
    "plannerVersion" : 1,
    "namespace" : "bla.my_collection",
    "indexFilterSet" : false,
    "parsedQuery" : {
        "foo" : {
            "$eq" : false
        }
    },
    "winningPlan" : {
        "stage" : "LIMIT",
        "limitAmount" : 100,
        "inputStage" : {
            "stage" : "SKIP",
            "skipAmount" : 9000,
            "inputStage" : {
                "stage" : "FETCH",
                "inputStage" : {
                    "stage" : "IXSCAN",
                    "keyPattern" : {
                        "foo" : 1,
                        "meta_data.created_at" : -1
                    },
                    "indexName" : "foo_1_meta_data.created_at_-1",
                    "isMultiKey" : false,
                    "multiKeyPaths" : {
                        "foo" : [ ],
                        "meta_data.created_at" : [ ]
                    },
                    "isUnique" : false,
                    "isSparse" : false,
                    "isPartial" : false,
                    "indexVersion" : 1,
                    "direction" : "forward",
                    "indexBounds" : {
                        "foo" : [
                            "[false, false]"
                        ],
                        "meta_data.created_at" : [
                            "[MaxKey, MinKey]"
                        ]
                    }
                }
            }
        }
    },

}

: , - ​​ . , ?

"rejectedPlans" : [
            {
                "stage" : "SKIP",
                "skipAmount" : 19000,
                "inputStage" : {
                    "stage" : "SORT",
                    "sortPattern" : {
                        "meta_data.created_at" : -1
                    },
                    "limitAmount" : 19100,
                    "inputStage" : {
                        "stage" : "SORT_KEY_GENERATOR",
                        "inputStage" : {
                            "stage" : "FETCH",
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "foo" : 1,
                                    "_id" : 1
                                },
                                "indexName" : "foo_1__id_1",
                                "isMultiKey" : false,
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "foo" : [
                                        "[false, false]"
                                    ],
                                    "_id" : [
                                        "[MinKey, MaxKey]"
                                    ]
                                }
                            }
                        }
                    }
                }
            }

. ? ?

Answer 1

, ?

, . , , , . " + ".

?

, , , , , , , , .

?

"" , () . , , , , , .