How to configure aws CLI on s3 cp with anonymous user

Question

I need to upload files recursively from the s3 bucket. Bucket s3 allows anonymous access.

How do I list files and upload them without providing an AWS passkey with an anonymous user?

My team:

aws s3 cp s3:// anonymous@big-data-benchmark /pavlo/text/tiny/rankings/uservisits uservisit --region us-east --recursive 

The aws team claims that:

Unable to locate credentials. You can configure credentials by running "aws Unable to locate credentials. You can configure credentials by running "aws configure"

Answer 1

you may have to provide access keys and a secret key, even if you do anonymous access. You do not see an anonymous option for AWS cli.

another way to do this is to get to the http endpoint and capture the files this way.

In your case: http://big-data-benchmark.s3.amazonaws.com

You will get XML listing all the keys in the bucket. You can retrieve keys and call requests for each. Not the fastest thing, but it will do its job.

For example: http://big-data-benchmark.s3.amazonaws.com/pavlo/sequence-snappy/5nodes/crawl/000741_0

to get the files hanging up should be enough. for xml parsing depending on what you like, you can go as lo-level as sed and as high level as the correct language.

hope this helps.

Answer 2

You can use the no-sign-request option

aws s3 cp s3:// anonymous@big-data-benchmark /pavlo/text/tiny/rankings/uservisits uservisit --region us-east --recursive --no-sign-request