Uploading large files with Django: how to do it?

Question

I need to load> = 20 GB of data using Django. Do I have to break the file into pieces and then upload it using some kind of checksum to maintain integrity, or does Django implicitly do this?

Would it be better if I use FTP instead of regular HTTP for such large files?

Answer 1

Django uses the so-called Upload Handlers to upload files and has the FILE_UPLOAD_MAX_MEMORY_SIZE parameter associated with it (the default value is 2.5Mb). Files smaller than this threshold will be processed in memory, large files will be transferred to a temporary file on disk. I have not tried uploading files larger than 1 GB yet, but I would expect that you can just use django without any problems.