How to import csv data into django model

Question

I have some CSV data, and I want to import into django models using an example CSV data:

1;"02-01-101101";"Worm Gear HRF 50";"Ratio 1 : 10";"input shaft, output shaft, direction A, color dark green"; 2;"02-01-101102";"Worm Gear HRF 50";"Ratio 1 : 20";"input shaft, output shaft, direction A, color dark green"; 3;"02-01-101103";"Worm Gear HRF 50";"Ratio 1 : 30";"input shaft, output shaft, direction A, color dark green"; 4;"02-01-101104";"Worm Gear HRF 50";"Ratio 1 : 40";"input shaft, output shaft, direction A, color dark green"; 5;"02-01-101105";"Worm Gear HRF 50";"Ratio 1 : 50";"input shaft, output shaft, direction A, color dark green"; 

I have a django model called Product. The product has fields such as name , description and price . I want something like this:

 product=Product() product.name = "Worm Gear HRF 70(02-01-101116)" product.description = "input shaft, output shaft, direction A, color dark green" product.price = 100 

Answer 1

You want to use the csv module, which is part of the python language, and you have to use the Django get_or_create method

  with open(path) as f: reader = csv.reader(f) for row in reader: _, created = Teacher.objects.get_or_create( first_name=row[0], last_name=row[1], middle_name=row[2], ) # creates a tuple of the new object or # current object and a boolean of if it was created 

In my example, the model teacher has three attributes first_name, last_name and middle_name.

Django documentation get_or_create method

Answer 2

If you want to use the library, a quick google search for csv and django shows two libraries - django-csvimport and django-adapters . Let them read what they say about themselves ...

• django-adapters :

A Django adapter is a tool that makes it easy to convert a CSV / XML file to a python object or instance of a django model.

• django-importcsv :

django-csvimport is a universal importer tool that allows you to download CSV files to fill data.

First you need to write a model to match the csv file, and the second is more like a command line importer, which represents a huge difference in how you work with them, and each of them is good for a different type of project.

So which one to use? It depends on which one will best suit your project in the long run.

However, you can also completely avoid the library by writing your own django script to import the csv file, something like a line (warning, pseudo-code forward):

 # open file & create csvreader import csv, yada yada yada # import the relevant model from myproject.models import Foo #loop: for line in csv file: line = parse line to a list # add some custom validation\parsing for some of the fields foo = Foo(fieldname1=line[1], fieldname2=line[2] ... etc. ) try: foo.save() except: # if the're a problem anywhere, you wanna know about it print "there was a problem with line", i 

It is super easy. Hell, you can do this interactively through the django shell if this is a one-time import. Just indicate what you want to do with your project, how many files you need to process, and then - if you decide to use the library, try to find out which one is best suited to your needs.

Answer 3

The csv Python library can do your parsing, and your code can translate them into Products() .

Answer 4

You can also use django-adapters

 >>> from adaptor.model import CsvModel >>> class MyCSvModel(CsvModel): ... name = CharField() ... age = IntegerField() ... length = FloatField() ... ... class Meta: ... delimiter = ";" 

You declare MyCsvModel, which will correspond to the CSV file, for example:

Anthony; 27; 1.75

To import a file or any iterable object, simply do:

 >>> my_csv_list = MyCsvModel.import_data(data = open("my_csv_file_name.csv")) >>> first_line = my_csv_list[0] >>> first_line.age 27 

Without an explicit declaration, data and columns are mapped in the same order:

 Anthony --> Column 0 --> Field 0 --> name 27 --> Column 1 --> Field 1 --> age 1.75 --> Column 2 --> Field 2 --> length 

Answer 5

something like that:

 f = open('data.txt', 'r') for line in f: line = line.split(';') product = Product() product.name = line[2] + '(' + line[1] + ')' product.description = line[4] product.price = '' #data is missing from file product.save() f.close() 

Answer 6

You can use the django-csv-importer package. http://pypi.python.org/pypi/django-csv-importer/0.1.1

It works as a django model

 MyCsvModel(CsvModel): field1 = IntegerField() field2 = CharField() etc class Meta: delimiter = ";" dbModel = Product 

And you just need: CsvModel.import_from_file ("my file")

This will automatically create your products.

Answer 7

Consider using the built-in Django deserializers. Django docs are well written and can help you get started. Consider converting data from csv to XML or JSON and using a deserializer to import data. If you do this from the command line (and not through a web request), the loaddata manage.py command will be especially useful.

Answer 8

Here's a django egg for him:

django-csvimport

Answer 9

You can try django-import-export . It has a nice admin integration, a preview of changes, can create, update, delete objects.

Answer 10

define a class in models.py and a function in it.

 class all_products(models.Model): def get_all_products(): items = [] with open('EXACT FILE PATH OF YOUR CSV FILE','r') as fp: # You can also put the relative path of csv file # with respect to the manage.py file reader1 = csv.reader(fp, delimiter=';') for value in reader1: items.append(value) return items 

You can access element i in the list as elements [i]

Answer 11

Use the Pandas library to create csv data data.
Name the fields either by including them in the first line of the csv file, or in the code using the dataframe column method.
Then create a list of model instances.
Finally, use the django method . Bulk_create () to send a list of model instances to the database table.

The read_csv function in pandas is great for reading csv files and gives you lots of options for skipping lines, skipping fields, etc.

 import pandas as pd tmp_data=pd.read_csv('file.csv',sep=';') #ensure fields are named~ID,Product_ID,Name,Ratio,Description #concatenate name and Product_id to make a new field a la Dr.Dee answer products = [ Product( name = tmp_data.ix[row]['Name'] description = tmp_data.ix[row]['Description'], price = tmp_data.ix[row]['price'], ) for row in tmp_data['ID'] ] Product.objects.bulk_create(products) 

I used mmrs151's answer, but saving each line (instance) was very slow, and any fields containing a separator character (even inside quotation marks) were not processed by open () - line.split (';').

Pandas has so many helpful caveats worth finding out.

Answer 12

If you are working with newer versions of Django (> 10) and don’t want to waste time defining a model. you can use the ogrinspect tool.

This will create a code definition for the model.

 python manage.py ogrinspect [/path/to/thecsv] Product 

The result will be a definition of the class (model). In this case, the model will be called Product. You need to copy this code to the models.py file.

Then you need to transfer (in the shell) the new product table with:

 python manage.py makemigrations python manage.py migrate 

More information here: https://docs.djangoproject.com/en/1.11/ref/contrib/gis/tutorial/

Note that this example was run for ESRI Shapefiles, but it works well with standard CSV files.

You can use pandas to swallow your data (in CSV format).

 import pandas as pd your_dataframe = pd.read_csv(path_to_csv) # Make a row iterator (this will go row by row) iter_data = your_dataframe.iterrows() 

Now each line should be converted to a dictionary and use this dict to instantiate your model (in this case, Product ())

 # python 2.x map(lambda (i,data) : Product.objects.create(**dict(data)),iter_data 

Done, check your database now.