Python: unpredictable memory error when downloading large files

Question

I wrote a python script that I use to download a large number of video files (50-400 MB each) from an HTTP server. So far, it has worked well on long download lists, but for some reason it rarely has a memory error.

The device has about 1 GB of RAM, but I do not think that it ever exceeded the amount of RAM when running this script.

I controlled the memory usage in the task manager and perfmon, and it always behaves the same from what I saw: it slowly increases at boot time and then returns to its normal level after the download is completed (there are no small leaks that creep in or something in like that).

The download method is that it creates a file that remains at 0 KB until the download is complete (or the program fails), then it immediately writes the entire file and closes it.

for i in range(len(urls)): if os.path.exists(folderName + '/' + filenames[i] + '.mov'): print 'File exists, continuing.' continue # Request the download page req = urllib2.Request(urls[i], headers = headers) sock = urllib2.urlopen(req) responseHeaders = sock.headers body = sock.read() sock.close() # Search the page for the download URL tmp = body.find('/getfile/') downloadSuffix = body[tmp:body.find('"', tmp)] downloadUrl = domain + downloadSuffix req = urllib2.Request(downloadUrl, headers = headers) print '%s Downloading %s, file %i of %i' % (time.ctime(), filenames[i], i+1, len(urls)) f = urllib2.urlopen(req) # Open our local file for writing, 'b' for binary file mode video_file = open(foldername + '/' + filenames[i] + '.mov', 'wb') # Write the downloaded data to the local file video_file.write(f.read()) ##### MemoryError: out of memory ##### video_file.close() print '%s Download complete!' % (time.ctime()) # Free up memory, in hopes of preventing memory errors del f del video_file 

Here is the stack trace:

  File "downloadVideos.py", line 159, in <module> main() File "downloadVideos.py", line 136, in main video_file.write(f.read()) File "c:\python27\lib\socket.py", line 358, in read buf.write(data) MemoryError: out of memory 

Answer 1

Your problem here: f.read() . This line tries to load the entire file into memory. Instead, read the chunks ( chunk = f.read(4096) ) and save the fragments in a temporary file.