Design Pattern for Real-Time Data

Question

I have a stream of events in real time. I am developing an interface to display this stream for the user, so that:

• The user will be able to search using one of the attributes present in the data.
• The user will be able to sort the data

Each event in the stream has a state associated with it, which can be changed at any point, so I continue to query the server for the last 10 minutes of data and change the state on the client side accordingly.

My first attempt was to use pagination. However, this causes the following problems:

• When a user views, say, the 15th page and a new event are added, I need to remember the current page the user is on and make sure that it does not change. Even if the newly added data is not pushed too much, this will affect the position of the event in the list, and therefore, the user must search where the previous event has moved.
• When the user sorts by a specific attribute, I need to re-sort the incoming data and display it. I am still facing the same problem as above.

Is there a design pattern for this interaction? Twitter uses endless scrolling . This template is awesome, but I'm not quite sure how to adapt it to situations:

• If the user can sort the data.
• Where I use the base database to provide data to the interface. How should pages be designed?

Any suggestions?

Answer 1

It seems that moving sorting to the forefront is the best way to achieve the desired result. I will not go into infinite scrolling / pagination, because there are probably better resources out there, but basically track the last line (or page) received and sent along with your request. Hope this helps other issues preventing you from realizing this.

In addition: depending on how real-time data you should probably look for other methods to get it, such as a long poll or WebSockets instead of redialing every X minutes.

Response format

{ 'status' : 'OK', 'last_row_retrieved': 20, 'events' : [{ 'html' : "<div>The regular response</div>", 'sort_attr_1' : 10, ... }] } 

Front end

 //Maintain a container for the results you get each call. var myevents = []; $.post(...,function(response) { var data = $.parseJSON(response); var events = data.events; for(var i=0;i<events.length;i++) { //Save each event myevents.push(events[i]); } DisplayEvents(); }); //When user tries to sort do so with custom sorting function myevents.sort(function(a,b) { return a.sort_attr_1 - b.sort_attr_2;}); DisplayEvents(); function DisplayEvents() { //loop through myevents and display them however you're doing it }