Send data from a database to a client without templates (node.js)

Question

Send data from a database to a client without templates (node.js)

Hi, I did some testing using Node.js vs Fulephp. I had a simple setup and tried to figure out which one was faster. I had 10 thousand entries in mogodb that were attracted to views. The setup was simple, no js, no css, with minimal html. I quickly noticed that setting up php was twice as fast. At first I fired nodejs as slower and moved my life. However, I decided to try node without jade, which I used as my template engine, and as a result of luck, I came across a column on stackoverflow that the philosophy behind jade is not so much speed as elegance. Then I decided to try node without any pace. engines. But I quickly ran into a problem, as I realized that I had no idea how to transfer data from the database and node to the client . I was on a long night of horror and despair. At some point, I came to the conclusion that I needed the help of socket.io. Although I managed to connect to socket.io, in the end I still could not figure out how to transfer data. Then I decided to return to using tempo. but this time I decided to try ejs. In the end, I managed to display some data that looked like [object Object] , but it was not 10 thousand records, more like 25. I decided to do the right thing and post my question here. I would like to visualize a view without templates to check if my assumptions are correct. In the end, I try to do two things to figure out how to pass data to the Node.js client form and see if it improves my performance.

Here is my app.js with comments:

/** * Mongo DB */ var mongous = require('mongous').Mongous, dbCollection = 'test.personnel'; /** * Module dependencies. */ var express = require('express'), app = module.exports = express.createServer(), pub = __dirname + '/public'; // Configuration app.configure(function(){ app.set('view options', {layout: false}); //not sure if i need these here, but left it in case app.use(app.router); app.use(express.static(pub)); }); //Simple templating //I took this example from stackoverflow, //can't find that post anymore, //though I can look if need be app.register('.html', { compile: function(str, options){ return function(locals){ return str; } } }); // Routes //This is where data is right now, it need to end up on the client side //This was working with jade and kinda worked with ejs (though I am not sure because I was getting [object Object]) //-------------------------------------------------- app.get('/', function(req, res){ mongous(dbCollection).find(function(output){ res.render('index.html',{ //the data is here, it works, i tested it with console.log data: output.documents }); }); }); //-------------------------------------------------- app.configure('production', function(){ app.use(express.errorHandler()); }); app.listen(3000); console.log('Express server listening on port %d in %s mode', app.address().port, app.settings.env);

and here is my opinion:

 <!DOCTYPE html> <html> <head> <meta charset='utf-8'> <title>Index</title> </head> <body> <div id="data"> Data needs to end up here. Somehow... </div> </body> </html>

As you can see, not so much. Now I understand that, most likely, I will need to use some kind of template engine on the client side, and as soon as I have the data on the client side, I can process it myself. Which may be even slower in the end, but my main goal is to understand how to transfer data in Node.js to the client so that I can continue to experiment. Please help if you can, this will improve my understanding of node significantly. Thanks.

EDIT: With the help of all of you, and especially josh3736, this is what I ended up with if you're interested ... http://pastie.org/private/z3fjjbjff8284pr2mafw

+4

javascript node.js

user000001 Feb 05 '12 at 10:38

source share

4 answers

If you are interested in a really crude performance test between PHP and Node.js, then you should consider blocking all middleware on the Node stack (e.g. Express.js and Mongous) for a direct approach using a native MongoDB driver and raw generation HTML Something like this (untested):

 var http = require('http') , mongo = require('mongodb') , db = new mongo.Db('foo', new mongo.Server('localhost', 27017, {})); db.open(function(err, db) { if (err) throw err; db.collection('people', function(err, collection) { var server = http.createServer(function(req, res) { res.writeHead('200', {'Content-Type': 'text/html'}); res.write('<html><head><title>Node.js Output</title></head>'); res.write('<body><div id="data">'); collection.find(function(err, cursor) { cursor.each(function(err, doc) { if (doc) { res.write('<div>' + doc.name + '</div>'); } else { res.end('</div></body></html>'); cursor.close(); } }); }); }).listen(8080, 'localhost'); console.log('OK: listening on http://localhost:8080/'); }); });

[Change] Updated use of the mongo driver to use the cursor instead of the "toArray" method.

The key choice here is the Node MongoDB driver; in particular, you want it to be as non-blocking as possible. Surprisingly, right now there is no perfect (IMHO).

0

maerics Feb 05 '12 at 23:22

source share

If you stick with an expression and some kind of database layer, use res.send() instead of res.render .

Also remember that node.js is brand new, so not every library is stable or fast because you use it in other languages. (For example: you could use a different approach to access mongo faster, or not use express.)

0

Theippo Feb 05 '12 at 23:30

source share

Well, this is the best I could come up with. This, of course, is based on Hippos's answer, but with a bit more material. However, now I am doing something really wrong, because now it looks even slower than when developing templates, much slower ...

The code in question:

 //-------------------------------------------------- app.get('/', function(req, res){ mongous(dbCollection).find(10000,function(output){ var o = output.documents, str = JSON.stringify(o); res.send(str); }); }); //--------------------------------------------------

Sigh...

0

user000001 Feb 07 '12 at 3:06

source share

josh3736 · Accepted Answer · 2012-02-07T04:48:40+0000

As you hint in your answer, part of the problem is the speed of the template engine itself; You found out that Jade is not the fastest - in fact, it is one of the slowest .

My favorite doT engine. In the performance test I am associated with, doT can display a template 5.4 million times per second. Jade can display a similar pattern at just 29,000 times per second. This is not even a competition.

However, the engine speed pattern is only a small part of the problem here. I believe that your real problem is that the Mongolian driver you are using is poorly designed for the asynchronous Node model. (Disclaimer: I never used Mongous, I just spent a few minutes looking at the code.)

Node is designed to work with data streams. In other words, you have to work on very small pieces of data at a time. In contrast, it seems that Mongous processes the entire data set and returns it to your code as a single JSON object.

This is convenient and great for small data sets, but it completely falls apart when working with large volumes of data such as you (10,000 records). Node will be completely blocked when analyzing and processing a large amount of data (which is very, very bad, because it will not be able to process any incoming connections), and the V8 memory management system is not optimized for large heap allocations like this.

To work with large datasets correctly, you need to use the Mongo driver, which transfers entries to your code, such as node-mongodb-native, or mongoskin , which simplifies the work with the API.

The maerics answer was on the right track, but it was wrong because it uses toArray , which poses the same problem as Mongous: the entire dataset maps to an array in memory. Instead, just use the each cursor method, which asynchronously calls your callback function for each returned record as it arrives. Thus, the entire result set is never in memory; you only work one at a time, allowing you to record records that you have already processed, and, if necessary, collect garbage.

Now that we have established how to get data from the database, we need to figure out how to get it for the client.

The problem is that the Express viewer expects that all your data will be accessible from the front so that the template engine can display a single line for sending to the client. As we discussed above, this is not such a good idea if you are dealing with thousands of records. The right way to do this is to transfer the data that we receive from Mongo directly to the client. Unfortunately, we cannot do this in Express view - they are not intended to be asynchronous.

Instead, you have to write a custom handler. You are already on this path with Hippo's answer and your own attempt, but you really need to use res.write() , not res.send . Like res.render , res.send expects you to get a complete response when you call it, because it internally calls res.end , ending the HTTP response. In contrast, res.write simply sends data over the network, leaving the HTTP response open and ready to send more data; in other words, the flow of your response. (Note that you must set the HTTP headers before starting streaming. For example, res.contentType('text/html'); )

Just because you manually process the response (the rendering system described above) does not prevent you from using the template engine. You can use the template for the header and footer of your document and one for each entry. Let use doT to put everything together.

First, declare our templates. In real life, you can download them from files (or even hack Express to download them for you as views and get the source of the template), but we will just declare them in our code.

 var header = doT.template('<!DOCTYPE html><html><head><title>{{=it.title}}</title></head><body>'); var record = doT.template('<p>{{=it.fieldname}}, ...</p>'); var footer = doT.template('</body></html>');

( doT.template returns a function that generates HTML from the above template, and an object that you pass to this returned function when it is called. For example, now we can call header({title:'Test'}); )

Now we are doing the real job in the request handler: (suppose we already have a collection from the Mongo driver)

 app.get('/', function(req, res){ res.contentType('text/html'); res.write(header({title:'Test'})); collection.find({}).each(function(err, doc) { if (err) return res.end('error! ' + err); // bail if there an error if (doc) { res.write(record(doc)); } else { // `doc` will be null if we've reached the end of the resultset. res.end(footer()); } }); });

Send data from a database to a client without templates (node.js)

More articles: