How to access keys from buckets with periods (.) In their names using boto3?

Question

Context

I am trying to get encryption status for all my buckets for a security report. However, since encryption is at a key level, I want to iterate through all the keys and get the general encryption status. For example, “yes” - all keys are encrypted, “no” if none are encrypted, and “partially” - some of them are encrypted.
I have to use boto3 because there is a known problem with boto where the encryption status for each key always returns None. See here.

Problem

I am trying to iterate over all the keys in each of my buckets using boto3. The following code works fine until it runs in buckets with names that contain periods, for example, "my.test.bucket".

from boto3.session import Session

session = Session(aws_access_key_id=<ACCESS_KEY>,
                  aws_secret_access_key=<SECRET_KEY>,
                  aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')

for bucket in s3_resource.buckets.all():
    for obj in bucket.objects.all():
        key = s3_resource.Object(bucket.name, obj.key)
        # Do some stuff with the key...

When it falls into a bucket with a period in the name, it throws this exception when called bucket.objects.all(), telling me to send all requests to a specific endpoint. This endpoint can be found in the release of the exception object.

for obj in bucket.objects.all():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 82, in __iter__
for page in self.pages():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 165, in pages
for page in pages:
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 85, in __iter__
response = self._make_request(current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 157, in _make_request
return self._method(**current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 395, in _make_api_call
raise ClientError(parsed_response, operation_name)botocore.exceptions.ClientError: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

Things i tried

• Setting the endpoint_url parameter to the endpoint of the bucket specified in the exception response, for example s3_resource = session.resource('s3', endpoint_url='my.test.bucket.s3.amazonaws.com')
• When specifying the area, the bucket is located as s3_resource = session.resource('s3', region_name='eu-west-1')

, qaru.site/questions/355822/... boto, , call_format s3Connection. , boto (. ).

Update

. , =).

from boto3.session import Session

session = Session(aws_access_key_id=<ACCESS_KEY>,
                  aws_secret_access_key=<SECRET_KEY>,
                  aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')

# First get all the bucket names
bucket_names = [bucket.name for bucket in s3_resource.buckets.all()]


for bucket_name in bucket_names:
    # Check each name for a "." and use a different resource if needed
    if "." in bucket_name:
        region = session.client('s3').get_bucket_location(Bucket=bucket_name)['LocationConstraint']
        resource = session.resource('s3', region_name=region)
    else:
        resource = s3_resource
    bucket = resource.Bucket(bucket_name)

    # Continue as usual using this resource
    for obj in bucket.objects.all():
        key = resource.Object(bucket.name, obj.key)
        # Do some stuff with the key...

Answer 1

github issues . . , S3 , .

FWIW :

s3.meta.client.get_bucket_location(Bucket='boto3.region')

Answer 2

, Ben.

import boto3
knownBucket = 'some.topLevel.BucketPath.withPeriods'
s3 = boto3.resource('s3')

#get region
region = s3.meta.client.get_bucket_location(Bucket=knownBucket)['LocationConstraint']

#set region in resource
s3 = boto3.resource('s3',region_name=region)