How to Abort All Incomplete Multipage Bucket Downloads

Question

Sometimes multi-user downloads freeze or don't populate for some reason. In this case, you are stuck with lost parts that are difficult to remove. You can list them with:

aws s3api list-multipart-uploads --bucket $BUCKETNAME 

I am looking for a way to interrupt them all.

Answer 1

Assuming you have your awscli setting and it will output JSON, you can use jq to project the necessary keys with:

 BUCKETNAME=<xxx> aws s3api list-multipart-uploads --bucket $BUCKETNAME \ | jq -r '.Uploads[] | "--key \"\(.Key)\" --upload-id \(.UploadId)"' \ | while read -r line; do eval "aws s3api abort-multipart-upload --bucket $BUCKETNAME $line"; done 

Answer 2

If you are performing a multi-page download, you can also create an S3 Management control panel.

a) Open the S3 bucket

b) Go to the Management tab

c) Click Add Lifecycle Rule

d) Now enter the name of the rule in the first step and select the Clear incomplete multi-page downloads check box. You now have the number of days to save the partial parts.

It. You can also see these steps in the attached screenshot.

Answer 3

You can configure lifecycle rules to automatically clear them after a while. Here's a blog post demonstrating how to do this in the console:

https://aws.amazon.com/blogs/aws/s3-lifecycle-management-update-support-for-multipart-uploads-and-delete-markers/

To do this in boto3:

 import boto3 s3 = boto3.client('s3') try: lifecycle = s3.get_bucket_lifecycle(Bucket='bucket') except ClientError: lifecycle = {'Rules': []} lifecycle['Rules'].append({ 'ID': 'PruneAbandonedMultipartUploads', 'Status': 'Enabled', 'Prefix': '', 'AbortIncompleteMultipartUpload': { 'DaysAfterInitiation': 7 } }) s3.put_bucket_lifecycle(Bucket='bucket', LifecycleConfiguration=lifecycle) 

Adding this configuration to cli would be the same:

 $ aws s3api get-bucket-lifecycle --bucket bucket > lifecycle.json # Edit the lifecycle, adding the same configuration as in the boto3 sample $ aws s3api put-bucket-lifecycle --bucket bucket --lifecycle-configuration file://lifecycle.json 

If your bucket does not have a lifecycle policy, get-bucket-lifecycle raises a ClientError value. A robust implementation will provide the correct error.

A policy with only this configuration will look like this:

 { "Rules": [ { "ID": "PruneAbandonedMultipartUpload", "Status": "Enabled", "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } } ] } 

Answer 4

Alternatively, you can use the Minio Client aka mc. It is open source and compatible with AWS S3.

List all part load in a linked bucket.

 $ mc ls -I s3/mybucketname 

To delete all incomplete downloads in the corresponding S3 bucket.

 $ mc rm -I -r --force s3/mybucketname 

I = incomplete r = recursive f = with force parameter

Hope this helps.

Disclaimer: I work for Minio .

Answer 5

Here is my oneliner that will interrupt ALL compound downloads regardless of status, provided that you have no spaces in your key / file name.

 BUCKETNAME=<xxx>;aws s3api list-multipart-uploads --bucket $BUCKETNAME --query 'Uploads[].[Key, UploadId]' --output text | awk '{print "aws s3api abort-multipart-upload --upload-id "$2" --bucket $BUCKETNAME --key " $1 " & wait"}{}' | bash