Convert Django QuerySet to pandas DataFrame

Question

I am going to convert a QuerySet Django to a pandas DataFrame as follows:

 qs = SomeModel.objects.select_related().filter(date__year=2012) q = qs.values('date', 'OtherField') df = pd.DataFrame.from_records(q) 

This works, but is there a more efficient way?

Answer 1

 import pandas as pd import datetime from myapp.models import BlogPost df = pd.DataFrame(list(BlogPost.objects.all().values())) df = pd.DataFrame(list(BlogPost.objects.filter(date__gte=datetime.datetime(2012, 5, 1)).values())) # limit which fields df = pd.DataFrame(list(BlogPost.objects.all().values('author', 'date', 'slug'))) 

Above was how I do the same. The most useful addition is to indicate which fields you are interested in. If this is just a subset of the available fields that interest you, then this will give a performance boost, I think.

Answer 2

Django Pandas solves this pretty neatly: https://github.com/chrisdev/django-pandas/

From README:

 class MyModel(models.Model): full_name = models.CharField(max_length=25) age = models.IntegerField() department = models.CharField(max_length=3) wage = models.FloatField() from django_pandas.io import read_frame qs = MyModel.objects.all() df = read_frame(qs) 

Answer 3

From the point of view of Django (I am not familiar with pandas ) this is normal. My only concern is that if you have a very large number of entries, you may run into memory problems. If that were the case, then something like this memory, an efficient query iterator , would be needed . (As a passage, as written, some rewriting may be required to allow the intelligent use of .values() ).

Answer 4

Maybe you can use model_to_dict

 import datetime from django.forms import model_to_dict pallobjs = [ model_to_dict(pallobj) for pallobj in PalletsManag.objects.filter(estado='APTO_PARA_VENTA')] df = pd.DataFrame(pallobjs) df.head()