Combining noSQL and ORM in an MVC environment for a real application

Question

I tried for some time to add some “cool” things that I read about noSQL (couchDB, mongoDB, Redis ...) in recent years in practical use.

I'm pretty used to writing apps with Django and started using Play! when Java is the only acceptable deployment option (and uses it). Both modules work, for example, with MongoDB , and django also has nonrel . But I never felt the need for noSQL.

Until I finally found what, in my opinion, was a good option for storing documents like MongoDB.

Use case

Let's say we need to manage the order and keep track of any complex items. Elements can have many different properties, for example. simplification, we could:

• Refrigerator that may have
  • one or two doors
  • be class A, B or C,
  • surface color
  • standalone or built-in
• an oven that may have:
  • gas or electricity or both
  • self-cleaning or not
  • standalone or built-in

SQL / ORM Solution

As you can see, each object can have several properties that can be limited by type.

In my regular DBMS, through ORM, I would go with the definition of the “product” model, then inherit two models: a refrigerator and an oven. If after some time the refrigerator receives another property, I change the model - and as such, the scheme, perform the migration and add the column.

noSQL solutions

noSQL solutions I can think of:

• using RDF (with something like Virtuoso or creating my own simplified triples repository)
• using a document-oriented db like MongoDB

Problem

But I don’t understand how another (simpler) development would be pragmatically switched to a noSQL solution still using the ORM framework with the right adapter (especially with DODB).

Let's say I use Django with MongoDB via mongodb-engine.

I still use the same ORM, so I still describe these objects as models, listing all the properties. Thus, ORM does exactly the same job! The cost of creating migration, if the model changes, is very limited by ORM (in particular, with something like the South), and does not require anything to independently study the new technology.

There may be / other / advantages for DODB and some specific to MongoDB (scalability, data processing, possibly performance), but ... what about the specific use case and problems that I describe?

Most likely, I will miss the point, so here is the question:

For this specific use case:

• Is this example good or bad for DODB (do you have a good one)?
• It would be wise to combine ORM for the base material (Users, Orders) and use noSQL without ORM for complex objects, is there any good reason to completely switch to noSQL or do I need to stay with ORM / SQL?

I understand that the answers to these questions may be partially subjective, so you can take on a full knowledge of the theory of noSQL and SQL and ORM stock; the presence of good bridges from exchange ORMs to the noSQL database. Suppose we are discussing this use case with MongoDB as an alternative to noSQL.

But there is a more general question, which is the main question of this SO post:

• Not a good ORM (e.g. JPA, ActiveRecord or Django), which makes noSQL and, in particular, less document-oriented databases?
• ... and is it worth using noSQL with the "classic" ORM?

(“little use” in terms of programming and maintenance, performance and similar criteria are another matter and require an accurate comparison of the product with the product).

[edit]

What I'm also trying to understand is that it is not better to refuse to use ORM when switching to noSQL. It would be nice to have more “dynamic” models, for example. I could write a table describing the models of refrigerators and ovens (fields), and the models of refrigerators and ovens in the code could dynamically build their representations (forms for editing and lists for displaying).

Related questions:

[edit] : here you can show my research, but also clarify that what I ask is not common for noSQL and SQL

• Is ORM redundant with NoSQL API? : seem to be! but I’m trying to understand why most frameworks (see above) provide access to noSQL through their ORMs. Is this a good idea or not?
• why using ORM with NoSql (e.g. MongoDB) : more specific than the previous one, but still the other way around. I affirm that ORMs make noSQL unusual , and not vice versa!
• When to replace RDBMS / ORM with NoSQL
• When to use MongoDB or other documentation-oriented systems?
• https://softwareengineering.stackexchange.com/questions/54373/when-would-someone-use-mongodb-or-similar-over-traditional-rdms

EDIT AND LINKS:

• Siena : The Java persistence API, inspired by the Google App Engine Python Datastore, is trying to draw a bridge between the SQL and NoSQL worlds.
• minimongo : lightweight, schematic, Pythonic object-oriented interface for MongoDB

Answer 1

This is what I get for stackoverflow trolling. Once an incomprehensible question is asked in a big question, and I have to offer my 2 cents (at the risk of my own project schedule).

I just finished a project in which I had to undo ORM from the model so that I could implement the NoSQL solution and found that it was not so difficult, although it was difficult to find the optimal approach from time to time. Therefore, without becoming too specific in my implementation, I will touch on what I had to do to make it work, as this may give some enlightenment when you travel along the same path.

My setup:

• Framework - Symfony 1.4
• ORM - Doctrine 1.4
• NoSQL is my own proprietary solution

Goal:

• Saving image paths in xml files and database
• Store html description paths in xml files and database

I did not want to store the images as drops in a permanent storage (database), and I did not want to store the image paths in the database, since I did not want to pay the overhead of creating a database connection and querying the path. Therefore, I decided to save the path information in NoSQL persistent storage (file system).

And to describe html, I did not want to create a text column in my table and store what could potentially be hundreds of html rows in the database, and the same reasons as above.

All my NoSQL files are related to an object (like a refrigerator). These files contain paths to their associated assets (description and html images), in what I call pointers that point to assets in the file system. I decided to use the XML format for storing data so that it looks something like this:

// Path to pointer file /home/files/app/needle/myApp/refrigerator/1/1.xml // Example pointer <pointer>/home/files/app/file/myApp/refrigerator/1.png</pointer> 

Now, within the framework, I had to redefine the save () methods so that I could save the above assets using the NoSQL API. It was quite simple, I just checked the parent calls and supported the values ​​included in the methods, so they would not break the logic chain (methods that call other methods with the same arguments) that I did not know about. I also added that custom NoSQL APIs throw exceptions because the main save () call was completed in a try / catch block. The only thing you need to be careful here is to determine if your NoSQL assets should stop the entire transaction. In my example, I had to figure out whether the image would be uploaded by storing the remaining form fields in the database (I decided to break the transaction).

I also had to modify load () methods to extract assets using the NoSQL API and standard model logic. As with conservation methods, doing this was not so difficult. I just needed to see what the parent classes do, and not guess with any argument values.

When everything was said and done, I was able to store images and html descriptions in the file system, with an XML file consisting of pointers pointing to their location. So now I don’t take a database call every time I need an asset.

Some considerations (they can be included in other NoSQL solutions, I had to write my own):

• You will not be able to request from refrigerators that have images from your permanent store. You will need to write some logic in your application in order to extract assets from NoSQL storage.
• Backups. When backing up your persistent storage data, you also need to back up NoSQL data.
• Orphans: Now that your scheme does not know about any assets that you may have, removing a line from your persistent storage will result in an orphan of the asset in the file system. So make sure your application has NoSQL storage cleanup logic when the row has been deleted.

I think that I encountered all the main obstacles that I encountered when implementing the NoSQL solution with ORM, if you have other questions, feel free to hit me.

- Change -

Replies to comments:

• As I said, I did not want to create a database connection and request only the path to the asset. I believe that it is better to use the NoSQL solution for this type of information, since there really is no reason to run queries against this type of information (images or HTML descriptions).

• Developing my own NoSQL solution was more of an ego problem. At work, there was a project to implement a custom NoSQL solution (which had a bad impression of MogileFS), and, frankly, is poorly designed and poorly implemented. But instead of just pointing out the bad, I challenged myself to offer a better solution, but for a side project. And due to the complex aspect, I did not study the existing NoSQL solutions, but looking back, I probably should have done it.

I still think that you can implement MongoDB or any NoSQL solution by redefining the crud functions with the model level of your ORM, is relatively easy. In fact, I not only implemented my NoSQL solution, but also added the ability to index data in SOLR (for full-text search) during crud functions, so anything is possible.

Answer 2

What I'm also trying to understand is that it would not be better to drop when using ORM when switching to noSQL

It is not very useful to completely abandon ORM. But you may have to rewrite it quite a bit. There are many little things, such as transactions, sequential writes, error handling, and data integrity checks that ORM can process for you in noSQL mode.

ORMs are not designed to handle all possible functions, even in SQL. They just do the “bulk” of the hard work. That is why ORM django provides direct access to the SQL class when you need it.

Answer 3

This is a fairly open question, there is no “right” answer. The decision between NoSQL and SQL (ORM) depends on too many factors. Some questions I would ask:

• How are you familiar with both technologies?
• In your scenario, what is the effect of trade-offs? The relational model provides some guarantees that NoSQL does not, and vice versa.
• How often does your model change? Application evolution usually requires model changes, but how many of them do you expect?

As I said, this is open. My personal suggestion is to start modeling using technology that you know. You can always integrate new components later if you really need it.

Of course, if the interest is “academic” for using NoSQL, don’t worry about optimal scenarios, use it, you will see what is good and bad about it.

EDIT on a comment (the answer does not fit in the comments area):

@Stefano. I'm afraid I don’t see your point of view, since using NoSQL in frameworks (or using ORM) depends on your needs.

This is not a “good use of this tool in this framework” problem, as support is (often) excellent. The problem should be "Should I use this tool, why and what benefits does it give me?"

If the answer is “yes, I need this because A, B and / or C”, then just keep using it.

If the answer is "no, because A or B" or "it does not matter," then either do not use it, or choose the option that you are most familiar with from the ones available to you.

That is, the fact that one infrastructure supports something does not mean that it is worse or better or should or should not be used. That is why I asked my questions. In the end, this is a question about NoSQL and SQL, because the tools you use to integrate it (ORM, SQL, whatever) are just a data access channel and less relevant than the storage system you choose for your problem ( since the tool will be limited by storage system by definition)