Is there a good basis for upgrading the MongoDB schema in Scala?

Question

What are the options for migrating / updating the MongoDB schema?

We (my colleagues and I) have a somewhat large (~ 100 million records) collection of MongoDB. This collection is mapped (ORM'd) to a Scala lift-mongodb that has gone through several different iterations. We have all kinds of code that processes missing fields, renames, deletes, transfers, etc.

While the whole “circuitless” thing can be nice and flexible, in this case it causes a lot of noise in the code as our object continues to evolve. The continuation of this path of "flexible object" is simply not sustainable.

How did you guys implement migration / upgrade schemes in MongoDB using Scala? Is there a framework for this? I know Foursquare uses Scala with MongoDB and Rogue (their own DSL queries) ... does anyone know how they handle their migrations?

Thanks.

Answer 1

Perhaps this may help somewhat, here is how Guardian.co.uk deals with this:

http://qconlondon.com/dl/qcon-london-2011/slides/MatthewWall_WhyIChoseMongoDBForGuardianCoUk.pdf

Schema Updates

This can be mitigated:

• Adding a Version Key to Each Document
• Version update every time the application changes the document
• Using MapReduce to force migration of documents from older versions, if required

Answer 2

I program MongoDB data migration with my own Scala "Subset" infrastructure. It allows you to easily define document fields, fine-tune serialization of data (for example, write "date" is a specific format, etc.), and also create queries and update modifiers in terms of certain fields. This "gist" gives a good idea