I have a node.js application using mongoDB and I have a function that is used to delete all documents in a collection and then re-populate the collections in my db with some sample documents.
I use this to test my application with known data once in a while when I write new features.
The problem I am facing is that if I call drop () for all my collections, then call some inserts to reload, sometimes I will have fewer records than I expect them to be inserted into the database, and sometimes whole collections will be absent, my counter checks that everyone says # records that correspond to the expected amount, but the whole collection may be missing, although it says that it was inserted.
I can only assume, since this happens in about 50% of cases, which, perhaps, insertions somehow start to be inserted before the drop () function completes, and then drop () destroys the inserted documents because it works async.
I use {safe: true} for all my insert / update commands, but db.collection.drop () does not accept any arguments and there seems to be no way to specify a safe option.
so I'm trying to use db.getLastError () after drop () to block inserts until the quote ends.
using db.getLastError () or db.getlasterror () throws an error: TypeError: Object # <Db> does not have a getlasterror method
simplified getLastError call:
var mongo = require("mongodb"); var db_conn = new mongo.Db(dbName, new mongo.Server(host, port, { auto_reconnect: true }), {}); db_conn.open(function(err, db) { if(!err) { console.log("Database connection: \033[32mopen\033[0m"); } else { console.log("Database connection: \033[31mfailed\033[0m"); } }); db_conn.getLastError(); this.db_conn = db_conn; this.users = db_conn.collection("users"); this.companies = db_conn.collection("companies"); this.suppliers = db_conn.collection("suppliers");
I tried this on the mongodb module versions 1.1.4 and 1.1.7 and just got the error.
Am I using the wrong link? I cannot find anything on stackoverflow or on the web at all on getlasterror in a sample code in JavaScript.
If there is a better way to do this, I am all ears.
js docs: http://docs.mongodb.org/manual/reference/javascript/#db.getLastError
mongo docs: http://www.mongodb.org/display/DOCS/getLastError+Command
---- UPDATE September 22, 2012
Using the answer below, I was able to debug a bit, what was the root problem, and as I expected. When the default connection pool consists of 5 open connections, drops drop off randomly, and then inserts follow. Depending on the order in which they are sent, and to the pools to which each one goes, if one drop ends quickly and then quickly inserts it before the drop for this collection ends, it will delete the inserted records later.
My fix was to wrap all drops of the chain sequence in the drop () command callback, so that every drop occurs after the callback for the previous drop was run, and then at the end of the call, a callback that disconnects all inserts. It's messy, but I tested the initializedb function about 25 times with a basic set of records from 20 simple inserts and loading 100,000 records. hasn't worked yet.
Time will tell, but this answer .lastError () worked to solve this problem.
---- UPDATE September 30, 2012
I wanted to test this for a week or so before replying to make sure that it really worked. I started my script initialization about 100 times in the last week, both on a basic set of 26 records and on sets of 100,000 records, which are all inserted in bulk. I have not seen how this happened using my new code.
Code I use to delete collections: (I have a DatabaseHandler object that I use with prototype functions for operations)
DatabaseHandler.prototype.dropAll = function(callback, callbackArg) { // drop all the documents var that = this; var finished = callback || function() { }; var finishedArg = callbackArg; this.companies.drop(function(err, reply) { if(reply) { console.log("[db] Companies collection dropped"); } else { console.log("\033[31m[Error]\033[0m Companies collection failed to drop"); } // drop all the user documents that.users.drop(function(err, reply) { if(reply) { console.log("[db] Users collection dropped"); } else { console.log("\033[31m[Error]\033[0m Users collection failed to drop"); } // drop all the course documents that.courses.drop(function(err, reply) { if(reply) { console.log("[db] Courses collection dropped"); } else { console.log("\033[31m[Error]\033[0m Courses collection failed to drop"); } // drop all the course purchase documents that.course_purchases.drop(function(err, reply) { if(reply) { console.log("[db] Course purchases collection dropped"); } else { console.log("\033[31m[Error]\033[0m Course purchases collection failed to drop"); } console.log("Dropped all documents and collections"); return finished(finishedArg); }); }); }); }); };
Then I have an initialization function that takes an argument to indicate whether I want to initialize additional load test data (addloadtestdata). if I specify dropdata, it calls dropAll and accepts the initialization function for the callback. therefore, when the quote is completed, it causes the callback to be initialized and begins to re-add documents only after it is completed. I also have a small counting function that checks how many documents are inserted after the completion of the insertion, as well as to verify the correctness of the entered number, and not one of them was missed. this is probably the best function that I added, so every time I initialize, I can see if the exact number of inserted documents was correct or if I was missing even 1.
function initialize(doLoadtest){ // do work here } // drop all collections and documents if(dropdata) { // drop all the data first then call the initialize as a callback function when completed dropping Database.dropAll(initialize, includeLoadtestData); } else { initialize(includeLoadtestData); }
hope that helps