Twisted, FTP and Streaming Files

Question

I am trying to implement what can best be called the "FTP interface for the HTTP API." Essentially, there is an existing REST API that can be used to manage user files for the site, and I am creating a reseller server that re-provides this API as an FTP server. Thus, you can log in, say, with Filezilla and list your files, upload new ones, delete old ones, etc.

I am trying to do this using twisted.protocols.ftpfor server (FTP) and twisted.web.clientfor client (HTTP).

The thing I come across is when a user tries to upload a file, "streaming" this file from an HTTP response to my FTP response. Similarly for download.

The easiest approach is to download the entire file from the HTTP server, then deploy and send the contents to the user. The problem is that any file can have a large number of gigabytes (I think disk images, ISO files, etc.). However, with this approach, the contents of the file will be stored in memory between the time that I download from the API and the time that I send to the user is not good.

So, my solution is to try to "sink" it - since I get chunks of data from the API HTTP response, I just want to deploy and send these fragments with the FTP user. It's simple.

For my "custom FTP function" I use a subclass ftp.FTPShell. The method of reading this openForReading,, returns Delayed, which starts with the implementation IReadFile.

(, ) " HTTP". fetch HTTP-, , , , .

, - HTTP FTP, - , ftp._FileReader, , , send ( , , .). , "" , HTTP.

, - ? ? ( )?

from twisted.web import client
import urlparse

class HTTPStreamer(client.HTTPPageGetter):
    def __init__(self):
        self.callbacks = []

    def addHandleResponsePartCallback(self, callback):
        self.callbacks.append(callback)

    def handleResponsePart(self, data):
        for cb in self.callbacks:
            cb(data)
        client.HTTPPageGetter.handleResponsePart(self, data)

class HTTPStreamerFactory(client.HTTPClientFactory):
    protocol = HTTPStreamer

    def __init__(self, *args, **kwargs):
        client.HTTPClientFactory.__init__(self, *args, **kwargs)
        self.callbacks = []

    def addChunkCallback(self, callback):
        self.callbacks.append(callback)

    def buildProtocol(self, addr):
        p = client.HTTPClientFactory.buildProtocol(self, addr)
        for cb in self.callbacks:
            p.addHandleResponsePartCallback(cb)
        return p

def fetch(url, callback):

    parsed = urlparse.urlsplit(url)

    f = HTTPStreamerFactory(parsed.path)
    f.addChunkCallback(callback)

    from twisted.internet import reactor
    reactor.connectTCP(parsed.hostname, parsed.port or 80, f)

, Twisted - Dave Peticolas Twisted Introduction, , .

, .

Answer 1

, - HTTP FTP, - , ftp._FileReader, , , send call ( , ..). , "" , HTTP.

ftp._FileReader -, , HTTPStreamer , . / HTTP, . HTTP, . - ...

class FTPStreamer(object):
    implements(IReadFile)

    def __init__(self, url):
        self.url = url

    def send(self, consumer):
        fetch(url, consumer.write)
        # You also need a Deferred to return here, so the 
        # FTP implementation knows when you're done.
        return someDeferred

/, , , HTTP- , FTP- .