All geek questions in one place

Does S3 support HTTP pipelining?

I have thousands of small files (about 1 KB) to upload to S3 every minute. If I upload each file in the cycle "send my HTTP request - wait for the S3 response HTTP - send the next request - wait for the next response ...", it cost a lot of time because I have to wait 2 times for latency between S3 and my server. Of course, I already use the Keep-Alive HTTP header.

So, I am trying to send several HTTP requests without waiting for the corresponding one (HTTP pipelining). I am trying to send 20 requests to a packet and wait for 20 responses. I expected this to save a lot of time because I can still send a request when the previous answer was on the way.

However, this does not make the world a better place.

I send my 20 requests after about 200 ms, and then try to get a response. I expected that I could get an answer as quickly as I send requests after receiving the first answer, for example this graph .

The facts are that after I got the first answer, I have to wait about 300 ms for each answer. It is not better to compare with sending one request and receiving one answer.

Why can't I shorten the time for conveyor technology? Why does S3 cost so much time for every request? Does S3 support HTTP pipelining?

Thanks.

+6
source share
2 answers

Amazon S3 sports concurrency to circumvent the latency problem for every request.

You can make hundreds of simultaneous requests to S3 and upload large batches of files in a very short period of time.

+4
source

According to this Ubuntu bug report , implementing HTTP pipelining in S3 violates the HTTP / 1.1 specification.

+2
source

Source: https://habr.com/ru/post/899210/

More articles:


All Articles