All geek questions in one place

Using aws credential profiles with the scala spark application

I would like to be able to use the ~ / .aws / credentials file, which I support with different profiles using my scala spark application, if possible. I know how to configure hadoop for s3a inside my application, but I don’t want to continue to use different hard-coded keys and rather just use my credential file as for different programs. I also experimented using a java api like this val credentials = new DefaultAWSCredentialsProviderChain().getCredentials()and then created an s3 client, but this does not allow me to use my keys when reading files from s3. I also know that keys can go in core-site.xmlwhen I launch my application, but how can I manage different keys, as well as how to configure it using IntelliJ so that I can use different keys using different profiles?

+4
source share
2 answers

DefaultAWSCredentialsProviderChain does not contain providers by default. You need to add some, for example:

val awsCredentials = new AWSCredentialsProviderChain(new 
  auth.EnvironmentVariableCredentialsProvider(), new 
  auth.profile.ProfileCredentialsProvider(), new 
  auth.AWSCredentialsProvider())

You can use them with an S3 client or, as you noticed by Spark:

hadoopConfig.set("fs.s3a.access.key", awsCredentials.getAWSAccessKeyId)
hadoopConfig.set("fs.s3a.secret.key", awsCredentials.getAWSSecretKey)

To switch between different AWS profiles, you can switch between profiles by setting the AWS_PROFILE environment variable. If you need to, expand to any point.

0
source

If you have the AWS_env vars suite installed, spark-submitcopy them as s3a secrets.

S3A, fs.s3a.aws.credentials.provider, , , . : , URI config, env vars , , IMS EC2. ( , ) ... , com.amazonaws.auth.AWSCredentialsProvider.

fs.s3a.aws.credentials.provider com.amazonaws.auth.profile.ProfileCredentialsProvider (, , . , ... , . , -.

0

Source: https://habr.com/ru/post/1649591/

More articles:


All Articles