Elasticsearch EC2 Discovery Does Not Perform with Invalid Credentials

Question

I am trying to create an Elasticsearch cluster on EC2 and get errors that I provided the wrong credentials, but these are the same credentials that I used when creating the instances via jclouds.

An example of the error log that I see from Elasticsearch:

[2014-03-03 21:32:26,109][INFO ][node                     ] [Baron Blood] version[1.0.1], pid[6832], build[5c03844/2014-02-25T15:52:53Z]
[2014-03-03 21:32:26,110][INFO ][node                     ] [Baron Blood] initializing ...
[2014-03-03 21:32:26,127][INFO ][plugins                  ] [Baron Blood] loaded [cloud-aws], sites []
[2014-03-03 21:32:30,736][INFO ][node                     ] [Baron Blood] initialized
[2014-03-03 21:32:30,736][INFO ][node                     ] [Baron Blood] starting ...
[2014-03-03 21:32:30,932][INFO ][transport                ] [Baron Blood] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.154.175.62:9300]}
[2014-03-03 21:32:31,228][WARN ][org.apache.http.impl.client.DefaultHttpClient] Authentication error: Unable to respond to any of these challenges: {}
[2014-03-03 21:32:31,388][INFO ][discovery.ec2            ] [Baron Blood] Exception while retrieving instance list from AWS API: AWS was not able to validate the provided access credentials
[2014-03-03 21:32:46,415][WARN ][org.apache.http.impl.client.DefaultHttpClient] Authentication error: Unable to respond to any of these challenges: {}
[2014-03-03 21:32:46,425][INFO ][discovery.ec2            ] [Baron Blood] Exception while retrieving instance list from AWS API: AWS was not able to validate the provided access credentials
[2014-03-03 21:33:00,939][WARN ][discovery                ] [Baron Blood] waited for 30s and no initial state was set by the discovery
[2014-03-03 21:33:00,939][INFO ][discovery                ] [Baron Blood] adstage-es-log/KolEM00zT9mYvYn3mDkrow
[2014-03-03 21:33:00,946][INFO ][http                     ] [Baron Blood] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.154.175.62:9200]}
[2014-03-03 21:33:00,998][INFO ][node                     ] [Baron Blood] started
[2014-03-03 21:33:01,454][WARN ][org.apache.http.impl.client.DefaultHttpClient] Authentication error: Unable to respond to any of these challenges: {}
[2014-03-03 21:33:01,463][INFO ][discovery.ec2            ] [Baron Blood] Exception while retrieving instance list from AWS API: AWS was not able to validate the provided access credentials
[2014-03-03 21:33:01,466][INFO ][cluster.service          ] [Baron Blood] new_master [Baron Blood][KolEM00zT9mYvYn3mDkrow][ip-10-154-175-62][inet[/10.154.175.62:9300]], reason: zen-disco-join (elected_as_master)
[2014-03-03 21:33:01,516][INFO ][gateway                  ] [Baron Blood] recovered [0] indices into cluster_state

My elasticsearch.yml file seems to be correct based on the documentation:

cluster.name: adstage-es-log
cloud.aws.access_key: MY_ACCESS_KEY
cloud.aws.secret_key: MY_SECRET_TOKEN
cloud.aws.region: us-east
discovery.type: ec2
discovery.ec2.ping_timeout: 30s

In addition, as someone asks, the security group is wide open in these fields. Using Elasticsearch 1.0.1 and ec2 plugin 2.0.0.RC1.

So far I have not found anything indicating what might be causing this. Any ideas on how to solve this problem?

Answer 1

, , AWS, IAM. , , :

• ec2: DescribeAvailabilityZones
• ec2: DescribeInstances
• ec2: DescribeRegions
• ec2: DescribeSecurityGroups
• ec2: DescribeTags

Answer 2

, :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1404060922000",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeRegions",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeTags"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Answer 3

To add a specific example to Gordon's helpful answer, here is the current IAM permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    }
  ]
}