Publish / subscribe with big data

Question

I have a very large collection (~ 40,000 documents with ~ 20-25 fields, including arrays with a set of ~ 500 elements) and ~ 2,000 subscribers (now they are just bots).

So this is really hard work on the client, when the user subscribes to the entire collection (excluding some fields in the server publication) and uses Collection.find with custom filters, sorting and ordering

I tried using a publication with parameters: i.e. client-defined filters, etc. But in this case, I have too many memory leaks on the server and epic fail :) after a few hours.

Can anyone elaborate on some publication-subscription schemes for such collections? I do not ask for a clear solution, but some useful thoughts

thanks

Answer 1

You might want to try https://github.com/alethes/meteor-pages

He has:

Incremental Subscriptions. Loads only what is needed, and not the entire collection at once. Suitable for large data sets.

Answer 2

DO NOT post all documents. Publish only the documents you want to show and use pagination. I wanted to write a paging tutorial in Meteor, but right now I don't have much time.

Use Iron Router and its waitOn for client subscribers.

Router.map(function () { this.route('postShow', { path: '/posts/:page', before: function() { Session.set('page', this.params.page); }, waitOn: function () { //how many documents we want to per page var perPage = 50; return Meteor.subscribe('posts', {}, { //limit limit: perPage, //we need to compute how many documents to skip skip: perPage * Session.get('page') - perPage }); }, data: function() { return { //variable posts available for Handlebars: {{#each posts}}... posts: Posts.find() } } }); }); 

And publish:

 Meteor.publish('posts', function(params, opts) { return Posts.find(params, opts); });