How to pass POST data to Python requests?

Question

I am using the Python library requeststo send a POST request. The part of the program that produces the POST data can be written to an arbitrary file object (output stream).

How can I make these two parts suitable?

I would suggest that it requestsprovides a streaming interface for this use case, but it doesn't seem to do it. It takes as an argument a datafile-like object from which it reads . It does not provide a file-like object to which I can write .

Is this a fundamental problem with Python HTTP libraries?

Ideas so far:

It seems like the easiest solution is fork()to have the request library interact with the POST data producer, h strong>.

Is there a better way?

Alternatively, I could try to complicate the POST data producer. However, it is parsing a single XML stream (from stdin) and creating a new XML stream for use as POST data. Then I have the same problem in the reverse order: XML serializer libraries want to write to a file-like object, I don’t know about the possibility that the XML serializer provides a file-like object from which another can read .

, , - , Python (yield). POST (yield) pull-parser.

requests POST? XML, yield?

- -, - / - , ?

Answer 1

request data, Chunk-Encoded Requests. , .

, queue.Queue - :

import requests
import queue
import threading

class WriteableQueue(queue.Queue):

    def write(self, data):
        # An empty string would be interpreted as EOF by the receiving server
        if data:
            self.put(data)

    def __iter__(self):
        return iter(self.get, None)

    def close(self):
        self.put(None)

# quesize can be limited in case producing is faster then streaming
q = WriteableQueue(100)

def post_request(iterable):
    r = requests.post("http://httpbin.org/post", data=iterable)
    print(r.text)

threading.Thread(target=post_request, args=(q,)).start()

# pass the queue to the serializer that writes to it ...    
q.write(b'1...')
q.write(b'2...')

# closing ends the request
q.close()

Answer 2

, , , - . "" "" - , , . parallelism , , , , . , , , OS.