Can I use Hadoop with AWS4-HMAC-SHA256?

Question

My new bucket uses ASC Signature Version 4 . I try to use it with Hadoop and I get a message:

Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error Message: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>InvalidRequest</Code> <Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message> </Error> 

This is not mentioned on the Amazon S3 Hadoop page. Is Hadoop incompatible with S3 now or have I missed the configuration option?

I tried adding -Dcom.amazonaws.services.s3.enableV4 , as suggested on the proposed on the SDK page , no luck. I assume Hadoop does not use the Amazon SDK.

FWIW I use Apache Spark, but it uses Hadoop.

EDIT: I found this Jira ticket .

Answer 1

You are probably trying to get data from s3n that won't work. Go to s3a and remember to enable the endpoint:

 hdfs dfs -Dfs.s3a.awsAccessKeyId=<access key ID> -Dfs.s3a.awsSecretAccessKey=<secret acces key> -Dfs.s3a.endpoint=<s3 enpoint> -ls s3a://<bucket_name>/... 

Endpoints can be found here: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region