Loading asynchronous files in Python

Question

I am trying to find a way to asynchronously load multiple files in Python (2.6), preferably through a query module. Gevent and Twisted will also be acceptable, as I will study them in the near future.

My application requires downloading 40+ files in a short period of time, I want to continuously download all 4 files at once. And every time the download of one file is completed, another begins, so it remains at 4. Is this possible?

Answer 1

You do not need to use any external library or framework for such a simple task, put the list of URLs in the queue, start 4 threads, and each thread must take an element from the queue and load it.

something like that:

import sys import os import urllib import threading from Queue import Queue class DownloadThread(threading.Thread): def __init__(self, queue, destfolder): super(DownloadThread, self).__init__() self.queue = queue self.destfolder = destfolder self.daemon = True def run(self): while True: url = self.queue.get() try: self.download_url(url) except Exception,e: print " Error: %s"%e self.queue.task_done() def download_url(self, url): # change it to a different way if you require name = url.split('/')[-1] dest = os.path.join(self.destfolder, name) print "[%s] Downloading %s -> %s"%(self.ident, url, dest) urllib.urlretrieve(url, dest) def download(urls, destfolder, numthreads=4): queue = Queue() for url in urls: queue.put(url) for i in range(numthreads): t = DownloadThread(queue, destfolder) t.start() queue.join() if __name__ == "__main__": download(sys.argv[1:], "/tmp") 

using:

 $ python download.py http://en.wikipedia.org/wiki/1 http://en.wikipedia.org/wiki/2 http://en.wikipedia.org/wiki/3 http://en.wikipedia.org/wiki/4 [4456497152] Downloading http://en.wikipedia.org/wiki/1 -> /tmp/1 [4457033728] Downloading http://en.wikipedia.org/wiki/2 -> /tmp/2 [4457701376] Downloading http://en.wikipedia.org/wiki/3 -> /tmp/3 [4458258432] Downloading http://en.wikipedia.org/wiki/4 -> /tmp/4