Boto generate_url for S3 PUT not working with IAM roles?

Question

I use the following Python / Boto code to create one-time URLs for uploading files to the Amazon S3 bucket:

from boto.s3.connection import S3Connection def get_signed_upload_url(): s3 = S3Connection(ACCESS_KEY_ID, SECRET_ACCESS_KEY, is_secure=True) return s3.generate_url(300, 'PUT', bucket=BUCKET, key=KEY, headers={'Content-Type': 'text/plain'}) 

He worked perfectly for several years, and he continues to work today.

But now I'm moving on to the IAM role, which will save me the hard coding of ACCESS_KEY_ID and SECRET_ACCESS_KEY . Therefore, I deleted the hard-coded keys, resulting in this code:

 from boto.s3.connection import S3Connection def get_signed_upload_url(): s3 = S3Connection(is_secure=True) return s3.generate_url(300, 'PUT', bucket=BUCKET, key=KEY, headers={'Content-Type': 'text/plain'}) 

With this code, whenever I create a URL and make a PUT request to it from my client web application (via Ajax), I get an HTTP 400 Bad Request error from S3:

 <Error> <Code>InvalidToken</Code> <Message>The provided token is malformed or otherwise invalid.</Message> <!-- account-specific stuff removed --> </Error> 

Why is this happening?

Additional information:

• This is deployed to an EC2 server, which automatically assigns its IAM role. I confirmed that the authorization from this role is correctly done for my Python code. All other parts of my code, including the generate_url() calls that generate the GET URLs, work fine, using the ACCESS_KEY_ID automatically passed to the EC2 instance environment.
• Python 2.7.9 and Boto 2.38.0.
• The IAM role has all the appropriate permissions, as evidenced by the fact that other parts of my code successfully interact with S3 (getting elements and creating elements) using the role. Only this particular PUT does not work.
• The interface logic sends the Content-Type header, so this header is there. I tried removing the Content-Type header from this Boto code and from the front-end logic, but the problem is still happening.

Answer 1

What is it for? I cannot reproduce this problem. I created a new instance of EC2, assigned it an IAM role in which AmazonS3FullAccess permission is allowed

 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "*" } ] } 

The earliest version of Python I could install was 2.7.12 with Boto 2.38.0

 [ root@ip-172-31-13-14 ~]# python -V Python 2.7.12 [ root@ip-172-31-13-14 ~]# pip freeze|grep boto boto==2.38.0 botocore==1.5.95 

Then I run the app.py code

 from boto.s3.connection import S3Connection def get_signed_upload_url(): BUCKET = 'test' KEY = 'test-2' s3 = S3Connection(is_secure=True) return s3.generate_url(300, 'PUT', bucket=BUCKET, key=KEY, headers={'Content-Type': 'text/plain'}) if __name__ == "__main__": url = get_signed_upload_url() print "curl -v -X PUT -H 'Content-Type: text/plain' -d @hello2.txt '" + url + "'" 

Then run the resulting cURL command and the file will load successfully.

 > Host: test.s3.amazonaws.com > User-Agent: curl/7.53.1 > Accept: */* > Content-Type: text/plain > Content-Length: 8585 > Expect: 100-continue > < HTTP/1.1 100 Continue * We are completely uploaded and fine < HTTP/1.1 200 OK < x-amz-id-2: redacted < x-amz-request-id: redacted < Date: Tue, 19 Dec 2017 05:45:15 GMT < x-amz-version-id: redacted < ETag: "redacted" < Content-Length: 0 < Server: AmazonS3 

At this point, I can only guess that perhaps the IAM role you are using does not have sufficient privileges to perform the PUT action? Otherwise, there may be a difference in how your client-side code actually executes the PUT request, but this is unlikely.

Other things to check:

• Verify that the time on the EC2 instance is accurate.
• Verify that the AWS_REGION environment variable is set

Answer 2

This is a permission issue when creating a signed URL. Can you provide the role that you have tied to the instance.

Try running this command explicitly in the EC to see exactly where it fails.

If this is not a problem with the role, then of course there is a problem with the boto SDK that you are using. It must be version 2.5.1 or later.

Have you tried to set the passkey and secret key as None?