Tastypie-nonrel, django, mongodb: too many nests

Question

I am developing a web application with django, backbone.js, tastypie and mongodb. To adapt tastypie and django to mongodb, I use django-mongodb-engine and tastypie-nerel. This application has a model project in which there is a list of tasks. So it looks like this:

class Project(models.Model): user = models.ForeignKey(User) tasks = ListField(EmbeddedModelField('Task'), null=True, blank=True) class Task(models.Model): title = models.CharField(max_length=200) 

Thanks to the tastypie-nerelle, getting a list of project tasks is done in a simple way by using a GET request in / api / v 1 / project /: id: / tasks /

Now I want to expand this task model with a list of comments:

  class Task(models.Model): title = models.CharField(max_length=200) comments = ListField(EmbeddedModelField('Comment'), null=True, blank=True) class Comment(models.Model): text = models.CharField(max_length=1000) owner = models.ForeignKey(User) 

The problem with this implementation is that tastypie-nonrel does not support another attachment, so it is not possible for a simple POST to comment on / api / v 1 / project /: id: / task /: id: / comments /

An alternative is to simply make a PUT request for task / api / v 1 / project /: id: / task /, but this will create problems if two users decide to add a comment to the same Task at the same time, since the last PUT will cancel previous.

The last option (in addition to changing tastypie-nonrel) is to not insert the Comment inside the Tasks and just keep the ForeignKey, so the request will go to / api / v 1 / Comment /. My question is, does this violate the benefits of using MongoDB (cross-query as needed)? Is there a better way to do this?

I have little experience with any of the stack technologies, so maybe I am not focusing on the problem. Any suggestions are welcome.

Answer 1

You seem to have nested too much. However, you can create your own methods / URL mappings for tastypie, and then run your own logic instead of relying on "autobus" tastipy. If you are concerned about a comment issue, you still need transactions. Then your code should be strong enough to handle the behavior of a failed transaction, for example, retry. This, of course, will greatly slow down your posts if you constantly block a large object with many authors, which also indicates a design problem.

One way to reduce this a bit is to write to an intermediate source, such as a task queue or redis, and then discard comments as necessary. It depends on how reliable / durable your decision is. The task queue should handle attempts to retry failed transactions at least; with redis you can do something with pub / sub.

You should consider a few things about your IMO design regarding MongoDB.

• Avoid creating too large monolithic objects. Although this is a Mongo advantage, it depends on your use. If, for example, you always return your project as a top-level object, then as tasks and comments grow, network traffic itself will kill performance.

  Imagine a very far-fetched example, in which a specific project data is 10 thousand, each task is 5 thousand, and each comment is 2 thousand, if you have a project with 5 tasks, 10 comments for the task, you are talking about 10k + 5 * 5k + 10 * 2k. For a very active project with a lot of comments, this will be a difficult post over the network. You can make slice / projection requests to come to terms with this problem, but with some limitations and consequences.

• A corollary of the above structures your objects in your use cases. If you do not need to collect things together, they can be in different collections. Just because you “think” that you need to bring them back together does not mean that a wise implementation should be obtained in the same selection (although this is usually ideal).

  Even if you need everything in one case / screen, another solution that may be possible in some projects is to load things in parallel or even put it off via JavaScript after loading the page using AJAX. For example, you can download task information at the top and then make an asynchronous call to download comments separately, similar to how Disqus or Livefyre work as integration on other sites. This can help solve the problem of your nesting somewhat as you get rid of the task / project level and just save some identifiers in each comment / entry to be able to query between collections.

• Remember that you cannot immediately receive all comments, and if you have a lot of comments, you may run into restrictions on the size of one document. Recently, the size is larger in modern versions of Mongo, however, as a rule, it does not make sense to have one record with a lot of data, returning to the first element above.

My recommendations:

• Use transactions if you are worried about losing comments.

• Add a task queue / redis / something lasting if you are worried about competing entries and lose things as a result of # 1. If not, ignore it. Is this the end of the world if you have lost a comment?

• Consider restructuring, in particular, comments in a separate collection to alleviate your talipia problems. Load items delayed or parallel if necessary.