Ruby MongodB - Improving Speed ​​with Multiple Collections

Question

I am using MongoDB with Ruby using mongogem.

I have the following script:

• for each document in the collection say coll1look key1andkey2
• searching for a document in another collection speaks coll2with corresponding values ​​for key1andkey2
• if there is a match, add the document extracted in # 2 with a new key key3whose value should be set to the value key3in the document specified in # 1
• insert updated hash into new collection coll3

A general recommendation with MongoDB was to handle cross-collection operations in application code.

So, I do the following:

    client = Mongo::Client.new([ '127.0.0.1:27017' ], :database => some_db, 
                               :server_selection_timeout => 5)
    cursor = client[:coll1].find({}, { :projection => {:_id => 0} }) # exclude _id
    cursor.each do |doc|
        doc_coll2 = client[:coll2].find('$and' => [{:key1 => doc[:key1]}, {:key2 => doc[:key2] }]).limit(1).first # no find_one method
        if(doc_coll2 && doc[:key3])
            doc_coll2[:key3] = doc[:key3]
            doc_coll2.delete(:_id) # remove key :_id
            client[:coll3].insert_one(doc_coll2)
        end
    end

, - 250 coll1 3600 (1 ) ~ 15000 , , , , .

? , ?

• Coll1

  {
      "_id" : ObjectId("588610ead0ae360cb815e55f"),
      "key1" : "115384042",
      "key2" : "276209",
      "key3" : "10101122317876"
  }
  

• Coll2

  {
      "_id" : ObjectId("788610ead0ae360def15e88e"),
      "key1" : "115384042",
      "key2" : "276209",
      "key4" : 10,
      "key5" : 4,
      "key6" : 0,
      "key7" : "false",
      "key8" : 0,
      "key9" : "false"
  }
  

• coll3

  {
      "_id" : ObjectId("788610ead0ae360def15e88e"),
      "key1" : "115384042",
      "key2" : "276209",
      "key3" : "10101122317876",
      "key4" : 10,
      "key5" : 4,
      "key6" : 0,
      "key7" : "false",
      "key8" : 0,
      "key9" : "false"
  }
  

Answer 1

:

• key1 $lookup
• $unwind
• doc, coll1.key2 == coll2.key2 $redact
• $project
• coll3 $out

:

db.coll1.aggregate([
    { "$lookup": { 
        "from": "coll2", 
        "localField": "key1", 
        "foreignField": "key1", 
        "as": "coll2_doc"
    }}, 
    { "$unwind": "$coll2_doc" },
    { "$redact": { 
        "$cond": [
            { "$eq": [ "$key2", "$coll2_doc.key2" ] }, 
            "$$KEEP", 
            "$$PRUNE"
        ]
    }}, 
    { 
      $project: {
         key1: 1, 
         key2: 1, 
         key3: 1, 
         key4: "$coll2_doc.key4",
         key5: "$coll2_doc.key5", 
         key6: "$coll2_doc.key6", 
         key7: "$coll2_doc.key7", 
         key8: "$coll2_doc.key8", 
     key9: "$coll2_doc.key9",  

      } 
    }, 
    {$out: "coll3"} 
], {allowDiskUse: true} );

db.coll3.find()

{
    "_id" : ObjectId("588610ead0ae360cb815e55f"),
    "key1" : "115384042",
    "key2" : "276209",
    "key3" : "10101122317876",
    "key4" : 10,
    "key5" : 4,
    "key6" : 0,
    "key7" : "false",
    "key8" : 0,
    "key9" : "false"
}

: MongoDB 3.4

$project, $addFields $replaceRoot, , MongoDB 3.4

:

db.coll1.aggregate([
    { "$lookup": { 
        "from": "coll2", 
        "localField": "key1", 
        "foreignField": "key1", 
        "as": "coll2_doc"
    }}, 
    { "$unwind": "$coll2_doc" },
    { "$redact": { 
        "$cond": [
            { "$eq": [ "$key2", "$coll2_doc.key2" ] }, 
            "$$KEEP", 
            "$$PRUNE"
        ]
    }}, 
    {$addFields: {"coll2_doc.key3": "$key3" }},
    {$replaceRoot: {newRoot: "$coll2_doc"}},
    {$out: "coll3"} 
], {allowDiskUse: true})

Answer 2

, - , , . .

, .

db.coll1.ensureIndex({"key1": 1, "key2": 1});
db.coll2.ensureIndex({"key1": 1, "key2": 1});

, 1/10xxxxxxth , .

, , find, .