Django repositories using boto - I can’t download mp3, but you can upload an image. In addition, suffering HTTP 307 pain

Question

We use the backendo boto (2.2.1) for django-storages (1.1.4) to upload files to the S3 bucket. It works great for images, but when I try to upload movie files (small mov, small avi) or mp3, I get a Broken pipe error.

This is strange.

Delving into a Django trace, I see the following exception:

boto.https_connection.InvalidCertificateException 

How can I use Cyberduck to inspect the bucket directly: sometimes it complains that I get a mismatch between the certificate for *.s3.amazonaws.com and the domain *.s3-external-3.amazonaws.com

In fact, logging shows that I am serving HTTP 307 temporary redirects. Is it possible that AWS sends some types of content one way and the other, but boto / something can't handle it? It seems that downloading movies seems to hit S3 twice, while images hit once, so it’s quite possible that boto is handling a 307 fine (and closed tickets for 307 support in boto are a couple of years old), so it could be fine, and something else.

But what? I left a pleasant, productive day in my head, and it is very frustrating.

Any suggestions on what might be and / or what to try to get around this?

(Note that this fails with boto S3 beta or with a simple S3 backend - it's just that boto gives me what looks like a more specific error)

Answer 1

I write this as an answer because it’s too long to fit into the comment. This does not answer your question, but it may help you get the answer.

The 307 redirect you receive happens because the bucket is in eu-west-1, but you get to the standard endpoint s3.amazonaws.com. S3 uses some DNS magic and HTTP redirects to route traffic from the S3 common endpoint to the correct regional endpoint.

To do this, most S3 clients use a subdomain link scheme that adds the byte name to the host name in the request. So, if you are trying to access your bucket, the host header in the request will be foofoofoo-bar.s3.amazonaws.com by default in boto, and then using DNS magic and HTTP redirection, S3 will eventually get your request for the right place . This should happen automatically in boto.

This approach may cause a problem if your bucket name includes "." because then the host header can be foofoofoo.bar.s3.amazonaws.com, and since the wildcard SSL certificate on the S3 endpoint is only suitable for one subdomain level, the period in the bucket name then leads to an SSL certificate verification error.

That's why I asked about the "." there is a bucket in your name, but apparently this is not a problem. Anyway, could you provide more context from the magazines? I would like to see what happens before the certificate validation error.