Usoop 1.x port list - 4 more unknown ports

Question

I configured and installed hasoop 1.2.1 single node. I configured the namenode and jobtracker addresses with ports as " hdfs: // localhost: 9000 " and " localhost: 9001 " respectively.

After starting the cluster ( start-all.sh ). After that, I ran netstat -nltp , which lists hadoop ports.

 50030 - jobtracker Web UI 50060 - tasktracker Web UI 50070 - namenode Web UI 50075 - datanode Web UI (http://localhost:50075/browseDirectory.jsp?dir=%2F) 50090 - secondary namenode Web UI 54310 - namenode (as configured in XML) 54311 - jobtracker (as configured in XML) 50010 - datanode (for data transfer) 50020 - datanode (for block metadata operations & recovery) 33447 - tasktracker ( not configured. Any unused local port is chosen by hadoop itself) 

But a couple of other ports were also busy, and this shows that this is a java process (I settled on chaos and confirmed that they belong only to this hadoop clan).

 48212 - ??? 41888 - ??? 47448 - ??? 52544 - ??? 

These are not fixed ports. They are selected dynamically. Because when I restarted the cluster ( stop-all.sh and start-all.sh ), the other ports were the same as the first time, except that these ports were changed

 48945 - tasktracker (This is fine, as explained before) 

What about other ports? What are these ports used for?

 44117 - ??? 59446 - ??? 52965 - ??? 56583 - ??? 

Answer 1

Thanks for posting this interesting question, Vivek.

I was very intrigued, and I dug up some code for Apache Hadoop 1.2.1 - the initial section for each of the master and slave; But there were no additional port bindings except the standard documented one.

I did a couple of experiments on ways to run namenode and monitor ports using netstat -nltpa

1) hasoop --config ../ conf namenode -regular

2) Direct call of the main class Namenode

3) Add default value core-default.xml and run namenode

My observation was for No. 2 and No. 3, only standard ports appeared, so I looked at java options, and that was bingo.

Comment all below in hadoop-env.sh and then run hasoop, you will see only the standard port, so the additional ports that you see are all JMX bin ports

 export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS" export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS" export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS" export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS" 

Hope this helps.

Answer 2

On a linux system, known services are usually listed in the /etc/services file. Here network utilities (e.g. netstat ) get friendly names for port numbers (e.g. 80 / http).

Some packages may update /etc/services . If the hadoop ports in hadoop have a dynamic range that is changing, then there would be no reason for this update.

References

http://www.cyberciti.biz/faq/find-out-which-service-listening-specific-port/
http://www.tldp.org/LDP/nag2/x-087-2-appl.services.html

Hope this helps.

Answer 3

 netstat -nltp 

Shows all active TCP, UDP, RAW, or Unix socket connections. Hadoop HDFS, Hbase, Zookeeper creates many sockets in between and uses for reading / writing or messaging.

The number of RPC Reader threads is created in org.apache.hadoop.hdfs.DFSClient and for reading / writing data from connections. hadoop.rpc.socket.factory.class.ClientProtocol will provide detailed information on how to create a socket / factory.