UnknownHostException from java application running in AWS ECS container docker on Fargate cluster

Question

I have a great Java application that I am trying to run on a fargate cluster in AWS. Image runs successfully on my local docker. When I run it in fargate, it starts successfully, but eventually it detects the following error, after which the application gets stuck:

! java.net.UnknownHostException: 690bd678bcf4: 690bd678bcf4: Name or service not known ! at java.net.InetAddress.getLocalHost(InetAddress.java:1505) ~[na:1.8.0_151] ! at tracelink.misc.SingletonTokenDBO$.<init>(SingletonTokenDBO.scala:34) ~[habari.jar:8.4-QUARTZ-SNAPSHOT] ! at tracelink.misc.SingletonTokenDBO$.<clinit>(SingletonTokenDBO.scala) ~[habari.jar:8.4-QUARTZ-SNAPSHOT] !... 10 common frames omitted Caused by: ! java.net.UnknownHostException: 690bd678bcf4: Name or service not known ! at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_151] ! at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[na:1.8.0_151] ! at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[na:1.8.0_151] ! at java.net.InetAddress.getLocalHost(InetAddress.java:1500) ~[na:1.8.0_151] !... 12 common frames omitted 

Scala code insult line:

  private val machineName = InetAddress.getLocalHost().getHostName() 

Some initial research suggests that the error is related to the contents of the / etc / hosts file in the container. Therefore, I created a small test program that demonstrates the same behavior as my real application, and also uploads the contents of / etc / hosts to stdout:

 import java.net.*; import java.io.*; public class NetworkTest { public static void main(String[] args) throws InterruptedException, IOException, FileNotFoundException { while(true) { networkDump(); Thread.sleep(10000); } } private static void networkDump() throws IOException, FileNotFoundException { System.out.println("/etc/hosts:"); System.out.println(""); FileReader f = new FileReader("/etc/hosts"); BufferedReader reader = new BufferedReader(f); String line = null; while((line = reader.readLine()) != null) { System.out.println(line); } System.out.println(""); dumpHostname(); } private static void dumpHostname() { try { String hostname = InetAddress.getLocalHost().getHostName(); System.out.printf("Hostname: %s\n\n", hostname); } catch(UnknownHostException e) { System.out.println(e.getMessage()); } } } 

Dockerfile:

 FROM openjdk:8 WORKDIR /site ADD . /site CMD ["java", "NetworkTest"] 

The result obtained from this in AWS is as follows:

 /etc/hosts: 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 3a5a4271a6e3: 3a5a4271a6e3: Name or service not known 

Compared to this output running in docker on my local machine:

 > docker run networktest /etc/hosts: 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.17.0.4 82691e2fb948 Hostname: 82691e2fb948 

A local version that does not receive an exception has an entry in / etc / hosts for the host name, and there is no entry for the host name in the AWS hosts file. I tried adding the /etc/rc.local file to manually add the hostname at the end of the localhost line, and simply adding the RUN command to the Docker file to do the same. None of them had any effect.

Does anyone know if there is a way to configure the image or the definition of an ECS task to properly configure the host name in AWS?

Answer 1

Pointing the hostname to 127.0.0.1 by going:

 echo "127.0.0.1 $HOSTNAME" >> /etc/hosts 

Fixed problem for me.

I am using Docker Compose. So I have a docker-compose.yml :

 version: '2' services: myservice: command: ["/set-hostname.sh", "--", "/run-service.sh"] 

and then the set-hostname.sh file looks like this:

 #!/bin/bash set -e shift cmd=" $@ " echo "127.0.0.1 $HOSTNAME" >> /etc/hosts exec $cmd 

Answer 2

So, I ran into exactly the same problem, and the fact is that, as you already mentioned, the host name does not make much sense. The only way to get the actual IP address of the instance that can be seen in VPC is to use the AWS task metadata API, which I did in my case. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint.html

I hooked up the following code to get the localhost IP address:

 try { final ResponseEntity<String> taskInfoResponse = this.restTemplate.getForEntity("http://169.254.170.2/v2/metadata", String.class); log.info("Got AWS task info: {}", taskInfoResponse); log.info("Got AWS task info: {}", taskInfoResponse.getBody()); if (taskInfoResponse.getStatusCode() == HttpStatus.OK) { try { final ObjectNode jsonNodes = this.objectMapper.readValue(taskInfoResponse.getBody(), ObjectNode.class); final JsonNode jsonNode = jsonNodes.get("Containers") .get(0).get("Networks") .get(0) .get("IPv4Addresses").get(0); log.info("Got IP to use: {}", jsonNode); if (jsonNode != null) { awsTaskInfo.setTaskAddress(InetAddress.getByName(jsonNode.asText())); } } catch (IOException e) { throw new IllegalArgumentException(e); } } else { awsTaskInfo.setTaskAddress(InetAddress.getLoopbackAddress()); } }catch (ResourceAccessException e){ log.error("Failed to fetch AWS info", e); awsTaskInfo.setTaskAddress(InetAddress.getLoopbackAddress()); } 

Answer 3

Exactly the same problem that I struggled with for a long time. This solution worked for me:

 ENTRYPOINT ["/bin/sh", "-c" , "echo 127.0.0.1 $HOSTNAME >> /etc/hosts && exec mvn spring-boot:run"]