Iterate over a large set of Django requests while data changes elsewhere

Question

Iterating over a set of queries, for example:

class Book(models.Model):
    # <snip some other stuff>
    activity = models.PositiveIntegerField(default=0)
    views = models.PositiveIntegerField(default=0)

    def calculate_statistics():
        self.activity = book.views * 4
        book.save()

def cron_job_calculate_all_book_statistics():
    for book in Book.objects.all():
        book.calculate_statistics()

... works just fine. However, this is a cron task. book.viewsincreases while this happens. If book.viewsmodified while this cronjob is running, it returns.

Now book.viewscronjob does not change, but it is cached during the request call .all(). When book.save(), I feel like he is using the old meaning book.views.

Is there a way to make sure that only the field is updated activity? As an alternative, let's say there are 100,000 books. This will take quite some time. But it book.viewswill be from the moment the request starts. Is the solution easy to use .iterator()?

: , . , , .

def calculate_statistics(self):
    self.activity = self.views + self.hearts.count() * 2
    # Can't do self.comments.count with a comments GenericRelation, because Comment uses
    # a TextField for object_pk, and that breaks the whole system. Lame.
    self.activity += Comment.objects.for_model(self).count() * 4
    self.save()

Answer 1

Django 1.1, :

from django.db.models import F
Book.objects.all().update(activity=F('views')*4)

:

for book in Book.objects.all().iterator():
    Book.objects.filter(pk=book.pk).update(activity=book.calculate_activity())

, , , , cron.

Answer 2

, , , iterator():

Book.objects.filter(stuff).order_by(stuff).iterator()

, Django ( ).

Answer 3

Regardless of how you solve this, be careful about transactional issues. For instance. The transaction isolation level is set to REPEATABLE READ by default, at least for the MySQL database. This is plus the fact that both the Django and db components work in a specific autocommit mode with the current transaction, that even if you use the (very nice) whrde clause, the value of `views' may be more invalid. I could be wrong here, but I feel warned.