All geek questions in one place

Upload a single file using multiple streams

I am trying to create a “Download Manager” for Linux that allows you to download a single file using multiple threads. This is what I am trying to do:

  • Separate the file to load into different parts, indicating the offset
  • Load the different parts in a temporary location.
  • Combine them into one file.

Steps 2 and 3 are resolvable, and at step # 1 I am stuck. How to specify the offset during file download?

Using something along the lines of open("/path/to/file", "wb").write(urllib2.urlopen(url).read()) , I cannot specify the starting point to read. Is there an alternative to this?

+4
source share
3 answers

To download part of a file, simply set the Range header as follows

 req = urllib2.Request(url) req.headers['Range'] = 'bytes=%s-%s' % (start, end) f = urllib2.urlopen(req)

However, not all servers support the Range header. Most file sharing services do not.

+3
source

first, the http server should return a Content-Length header. this usually means that the file is a static file, if it is a dynamic file, for example the result of php or jsp, you cannot make such a split.

then you can use the http Range header when requesting, this header tells the server which part of the file should be returned. see the python document on how to set and parse an HTTP header.

to do this, if the part size is 100 thousand, you first ask with the range: 0-1000000 100k will receive the first part, and in its length of the response it will report the file size, then start some stream with different Range, it will work

+3
source

See the .seek file at http://docs.python.org/library/stdtypes.html#file-objects .

This can do the trick.

Of interest, what is the reason for splitting the file?

0
source

Source: https://habr.com/ru/post/1401363/

More articles:


All Articles