How to handle external insertion into the Meteor database?

Question

We started using Meteor this week and really want to use it for as many projects as we can in the future.

Despite the fact that this afternoon we fall into the rock, in our ecosystem, we should be able to process and insert data inside mongodb from outside node / meteor.

Our two main solutions: MapReduce in an Hbase cluster and python scripts that parse CSV files and embed the results in MongoDB.

Right now, the only way to use data from MongoDB inside my Meteor application is when the data has been added through Meteor (console or form).

Let's take a quick example, I want to have access to the message via _id. Iron Router works great with this and gives helpers the ability to create a URL. With the same helpers, I could find two different results from three different uses.

The ObjectId seems to be wrong, and these are the three cases I encountered earlier:

http://0.0.0.0:3000/post/PAXEqRBB7RiNrdTTT => Inserted by Meteor, works fine http://0.0.0.0:3000/post/ObjectId(526fe0701d41c894b9105bff) => Inserted by python, broken http://0.0.0.0:3000/post/ObjectId(526fe0701d41c894b8715bff) => Inserted with meteor mongo shell 

Thus, I cannot access anything embedded in MongoDB with anything other than a meteor.

I found some related issues on Github, the main of which:

https://github.com/meteor/meteor/issues/61

But it was closed 8 months ago because the correction was on the roadmap. I am using the latest meteor (0.6.6.2), and this does not seem to be fixed yet.

My question is, what workaround can I find? I can't call JS methods like generating Meteor ObjectIDs inside Python, and what would be the best solution?

Should I use a node DDP application to handle all my external inserts in MongoDB?

Answer 1

If you want to interact directly with the database, I would not advocate the approach you take, which seems to be basically to create an API to interact with the database using middleware.

I think you have two options.

1. Make changes directly to MongoDB.

This is the easiest and fastest option, given that MongoDB bindings for almost every language. Changes you make to Mongo will be immediately selected by the Meteor oplog to monitor the driver if you installed Meteor with the Mongo oplog.

2. Talk to the Meteor server using DDP.

This is probably the “right” way to do something. It also has the advantage of allowing you to make RPC calls on the server, rather than just making changes to the database, so you can use all the Meteor.methods that you have already defined in your application.

See here for a really simple example of a DDP PDP client. You will need to work a bit to get the full reactivity that the Javascript client will see.

Update : Meteor itself now has a full-fledged DDP client, and one of the ways to launch it is as part of the usual Meteor server on Node. You can do whatever a normal browser client would do. For example, see https://github.com/VirtualLab/cooperation-loadtest .

Answer 2

Meteor overrides the default _id creation. The string option allows these identifiers to interact more easily when interacting with the console. I prefer that. You can have your external inserts include the identifier as a string or set your meteor env to use object identifiers.

http://docs.meteor.com/#meteor_collection

 ... idGeneration String The method of generating the _id fields of new documents in this collection. Possible values: 'STRING': random strings 'MONGO': random Meteor.Collection.ObjectID values