Boto3 S3, last sorted bucket

Question

I need to get a list of items from S3 using Boto3, but instead of returning the default sort order (descending), I want it to return it in the reverse order.

I know that you can do this through awscli:

aws s3api list-objects --bucket mybucketfoo --query "reverse(sort_by(Contents,&LastModified))"

and its feasibility through the user interface console (not sure if this is done on the client side or server side)

I don't seem to see how to do this in Boto3.

Currently, I am extracting all the files and then sorting ... but this seems redundant, especially if I only care about the 10 most recent files.

The filter system seems to accept only the prefix for s3, nothing more.

Answer 1

I made a small change to what @helloV posted below. its not 100% optimal, but it does work with the limitations that boto3 currently has.

s3 = boto3.resource('s3')
my_bucket = s3.Bucket('myBucket')
unsorted = []
for file in my_bucket.objects.filter():
   unsorted.append(file)

files = [obj.key for obj in sorted(unsorted, key=get_last_modified, 
    reverse=True)][0:9]

Answer 2

If there are not many objects in the bucket, you can use Python to sort as you wish.

Define lambda to get the last modified time:

get_last_modified = lambda obj: int(obj['LastModified'].strftime('%s'))

Get all objects and sort them by last modified time.

s3 = boto3.client('s3')
objs = s3.list_objects_v2(Bucket='my_bucket')['Contents']
[obj['Key'] for obj in sorted(objs, key=get_last_modified)]

If you want to cancel sorting:

[obj['Key'] for obj in sorted(objs, key=get_last_modified, reverse=True)]

Answer 3

, boto3. , boto3 :

all(), filter(**kwargs), page_size(**kwargs), limit(**kwargs)

. https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.ServiceResource.buckets

Answer 4

keys = []

kwargs = {'Bucket': 'my_bucket'}
while True:
    resp = s3.list_objects_v2(**kwargs)
    for obj in resp['Contents']:
        keys.append(obj['Key'])

    try:
        kwargs['ContinuationToken'] = resp['NextContinuationToken']
    except KeyError:
        break

Answer 5

. aws s3.

: aws s3 ls testing1-goreplay --recursive

, .

Answer 6

s3 = boto3.client('s3')

get_last_modified = lambda obj: int(obj['LastModified'].strftime('%Y%m%d%H%M%S'))

def sortFindLatest(bucket_name):
    resp = s3.list_objects(Bucket=bucket_name)
    if 'Contents' in resp:
        objs = resp['Contents']
        files = sorted(objs, key=get_last_modified)
        for key in files:
            file = key['Key']
            cx = s3.get_object(Bucket=bucket_name, Key=file)

, . Python3 AWS . . , . , "reverse = True".