How to delete or clean old files on S3?

Question

Are there existing solutions for deleting files older than x days?

Answer 1

Amazon recently introduced an item expiration .

Amazon S3 Announces Object Expiration

Amazon S3 has announced a new feature called Object Expiration, which allows you to plan the removal of your objects after a specified period of time. Using the object’s expiration date to schedule periodic removal of objects eliminates the need for you to identify objects to delete and send removal requests to Amazon S3.

You can define object expiration rules for a set of objects in your bucket. Each expiration rule for an object allows you to specify a prefix and expiration date in days. A prefix field (for example, logs/ ) identifies the object (s) subject to the expiration rule, and the expiration period indicates the number of days from the date of creation (i.e. age), after which the object should be deleted. As soon as objects pass their validity period, they will be queued for deletion. You will not be billed for storage for objects with an expiration date.

Answer 2

Here is some information on how to do this ...

http://docs.amazonwebservices.com/AmazonS3/latest/dev/ObjectExpiration.html

Hope this helps.

Answer 3

You can use AWS S3 lifecycle rules to expire and delete files. All you have to do is select the bucket, click on the Add Lifecycle Rules button and configure it, and AWS will take care of them for you.

You can link to Joe's blog post below for step-by-step instructions. It is pretty simple:

https://www.joe0.com/2017/05/24/amazon-s3-how-to-delete-files-older-than-x-days/

Hope it helps!

Answer 4

You can use the following Powershell script to delete an object that expires in x days .

 [CmdletBinding()] Param( [Parameter(Mandatory=$True)] [string]$BUCKET_NAME, #Name of the Bucket [Parameter(Mandatory=$True)] [string]$OBJ_PATH, #Key prefix of s3 object (directory path) [Parameter(Mandatory=$True)] [string]$EXPIRY_DAYS #Number of days to expire ) $CURRENT_DATE = Get-Date $OBJECTS = Get-S3Object $BUCKET_NAME -KeyPrefix $OBJ_PATH Foreach($OBJ in $OBJECTS){ IF($OBJ.key -ne $OBJ_PATH){ IF(($CURRENT_DATE - $OBJ.LastModified).Days -le $EXPIRY_DAYS){ Write-Host "Deleting Object= " $OBJ.key Remove-S3Object -BucketName $BUCKET_NAME -Key $OBJ.Key -Force } } }