Nodejs - chunked twitter json parsing

Question

The nodejs server receives this JSON stream from Twitter and sends it to the client:

stream.twitter.com/1/statuses/filter.json?track=gadget 

The data returned to the client is "chunked" JSON, and both JSON.parse (chunk) and eval ("+ chunk + ')") on the client side lead to parsing errors. Concatenating pieces and waiting for the end event is not a solution either

I noticed that the previous samples used something like this on the client side, which apparently worked before then:

  socket.onmessage = function(chunk) { data = eval("(" + chunk.data + ")"); alert(data.user.screen_name); 

I use this on the client side and this leads to a parsing error:

 var socket = new io.Socket(); socket.on('message', function(chunk) { var data = eval('(' + chunk + ')'); // parsing error alert(data.screen_name): 

I know that it successfully returns a JSON piece with:

  var socket = new io.Socket(); socket.on('message', function(chunk) { alert(chunk): // shows a JSON chunk 

Server:

  response.on('data', function (chunk) { client.each(function(e) { e.send(chunk); }); 

Has something changed or what else am I doing wrong?

UPDATE: The event "end" does not fire because its streaming?

 http.get({ headers: { 'content-type': 'application/json' }, host: 'stream.twitter.com', path: '/1/statuses/filter.json?track... }, function(res) { res.setEncoding('utf8'); res.on('data', function (chunk) { client.each(function(e) { e.send(chunk); }); }); // does not fire res.on('end', function () { }); ... 

I am studying the difference with http 1.0 and http 1.1 regarding sending data with channels.

Answer 1

See the Score Analysis section of the Twitter documentation.

Parsing JSON responses from the Streaming API is simple, each object is returned on a separate line and ends with a carriage return. Newline characters (\ n) can occur in object elements (for example, a text element of a state object), but carriage returns (\ r) should not.

On the server side, continue to accumulate chunks until you see the carriage return "\r" . As soon as the carriage returns, extract the line before the carriage returns, and this will give us one tweet.

 var message = ""; // variable that collects chunks var tweetSeparator = "\r"; res.on('data', function(chunk) { message += chunk; var tweetSeparatorIndex = message.indexOf(tweetSeparator); var didFindTweet = tweetSeparatorIndex != -1; if (didFindTweet) { var tweet = message.slice(0, tweetSeparatorIndex); clients.forEach(function(client) { client.send(tweet); }); message = message.slice(tweetSeparatorIndex + 1); } }); 

The client becomes simple. Just parse the socket message as JSON as a whole.

 socket.on('message', function(data) { var tweet = JSON.parse(data); }); 

Answer 2

@Anurag I'cant adds comments, however instead

 if (chunk.substr("-1") == "\r") 

it should be:

 if ( chunk.charCodeAt(chunk.length-2) == 13 ) 

Carriage Return is not the last character.

Answer 3

I would recommend passing the response to the JSON parser. You can use this: https://github.com/dominictarr/JSONStream