Hadoop tracking is only available with localhost

Question

I am setting up Hadoop (0.20.2). For starters, I just want it to work on the same machine — I will probably need a cluster at some point, but I will worry about it when I get there. I got this to the extent that my client code can connect to the job tracker and run the job, but there is one problem: the job tracker is only available from the same machine it is running on. I actually performed a port scan using nmap, and it shows port 9001, open when scanning from a Hadoop machine, and closes when it is from somewhere else.

I tried this on three machines (one Mac, one Ubuntu and a Ubuntu VM running in VirtualBox), the same thing. None of them have firewalls, so I'm sure this is a Hadoop problem. Any suggestions?

Answer 1

In the configuration files hadoop fs.default.name and mapred.job.tracker refer to localhost?

If so, then Hadoop will only listen on port 9000 and 9001 on the loopback interface, which is not available for any other host. Make sure that fs.default.name and mapred.job.tracker refer to the external hostname of your computer.

Answer 2

In addition to the above answer, I found that in /etc/hosts there was a line on the main (running ubuntu):

127.0.1.1 master

This meant that running nslookup master on the main returned the local address - therefore, despite using master in mapred-site.xml , I had the same problem. My solution (probably better) was to create an alias on my DNS server and use it instead. I think you can probably also change the IP address in /etc/hosts to an external one, but I have not tried this - I'm not sure what consequences it can have for other services.

Answer 3

Make sure you do not specify your wizard twice in the /etc/hosts . I had the following, which allowed the master to listen only 127.0.1.1

 127.0.1.1 hostname master 192.168.xx hostname master 192.168.xx slave-1 192.168.xx slave-2 

The above answer caused the problem. I changed my /ect/hosts to the following to make it work.

 127.0.1.1 hostname 192.168.xx hostname master 192.168.xx slave-1 192.168.xx slave-2 

Use netstat -an | grep :9000 netstat -an | grep :9000 to make sure your connections are working!