Node.js requests accidentally start to hang and will not be cleared until the server reboots

Question

I ran into a really strange and seemingly random problem in our web application that I simply cannot debug successfully. It works fine for 10 minutes to 6 hours, and then all of a sudden no remote requests to or from the server can be made, they just hang (this includes normal HTTP requests and web sockets). It is strange that visiting the site works regularly until the limit of the file descriptor of the OS is reached, and then http will fully work with all the blocked connections.

There are no errors, but when a problem occurs, the following error occurs (I assume that this is a side effect of what is happening, not the cause).

TypeError: Cannot read property '0' of null at null.<anonymous> (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/collection.js:504:22) at args.(anonymous function) (/app/node_modules/strong-agent/lib/proxy.js:85:18) at g (events.js:175:14) at EventEmitter.emit (events.js:98:17) at Base.__executeAllServerSpecificErrorCallbacks (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/base.js:315:29) at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:273:22 at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:370:11 at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:352:28 at _callback (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/db.js:670:5) at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/auth/mongodb_cr.js:47:13 

I tried to increase the file descriptor limits and the maxSockets global agents without affecting this behavior. At the same time, there is no traffic inflow, and this happens equally often during peak and off-peak periods. CPU usage stays consistently below 5% and does not have any noticeable changes that result in or during a failure. The server also never drops below 1 GB of free memory.

Stack: Cloud SmartOS (Joyent) server, Express, Socket.io, MongoDB and Redis.

I debugged this for several days and completely ran out of ideas where to look. Hoping that someone on SO came across something similar or have different ideas on what to try or check.

Answer 1

After countless hours of debugging and extra debugging, I finally found the culprit. The error was caused by several different mongojs callbacks that seem to have activated and blocked the connections from closing. Over time, this reached a tipping point, and the connections began to hang until the file descriptor limit was reached.

The error was in the module Now.js node (which was left). If there is someone who is facing this problem using Now.js, I forked it and fixed the error. You can find the commit here: https://github.com/goldfire/now/commit/b5bd54f8950602f752a710c606be6754b759cab2 .

As I found this error, I need to attach the error listener to the database object:

 var db = require('mongojs').connect('...', ['collection']); db.client.on('error', function(err){ console.log(err.stack); });