I have separate Docker images (1.9.1) for Nanoop (2.7.1) named datanode and datanode. I can create containers from them and transfer them through the user-defined Docker network. However, the datapath appears to be reporting that it has the IP address of a network gateway, not its own IP address. Although this does not cause any problems with a single datanode, the confusion reigns in adding additional datanodes. All of them are registered with the same IP address, and a namenode is displayed between them, and it is only ever reported that one datanode is alive.
Why does the server (namenode) read the wrong IP address from the client socket connection (datanode) when working on the Docker user network and how to fix it?
Update: this issue seems to be on the Docker side
Starting two containers with --net=bridgeand running netcat server:
nc -v -l 9000
in one container and netcat client in another:
nc 172.17.0.2 9000
causes the first container to print correctly:
Connection from 172.17.0.3 port 9000 [tcp / 9000] accepted
But creating a user network:
sudo docker network create
and executing the same commands in containers started with does --net=testnot correctly print the IP address of the network interface / user network interface:
Connection from 172.18.0.1 port 9000 [tcp / 9000] accepted
HDFS / Docker Details
The property dfs.datanode.addressin each datanode file hdfs-site.xmlhas its own host name (for example, hdfs-datanode-1).
The network is created as follows:
sudo docker network create
Nameday begins as follows:
sudo docker run -d \
--name hdfs-namenode \
-v /hdfs/name:/hdfs-name \
--net=hadoop-network \
--hostname hdfs-namenode \
-p 50070:50070 \
hadoop:namenode
And the datanode started like this:
sudo docker run -d \
--name hdfs-datanode-1 \
-v /hdfs/data_1:/hdfs-data \
--net=hadoop-network \
--hostname=hdfs-datanode-1 \
--restart=always \
hadoop:datanode
( sudo docker exec hdfs-namenode hdfs dfsadmin -report) :
...
Live datanodes (1):
Name: 172.18.0.1:50010 (172.18.0.1)
Hostname: hdfs-datanode-1
...
:
sudo docker exec hdfs-namenode cat /etc/hosts
, namenode , 172.18.0.2, datanode 172.18.0.3:
172.18.0.2 hdfs-namenode
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.3 hdfs-datanode-1
172.18.0.3 hdfs-datanode-1.hadoop-network
datanode :
172.18.0.3 hdfs-datanode-1
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.2 hdfs-namenode
172.18.0.2 hdfs-namenode.hadoop-network
ip route :
sudo docker exec hdfs-namenode ip route
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2
sudo docker exec hdfs-datanode-1 ip route
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.3
, datanode, namenode IP- datanode 172.18.0.1:
... INFO hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(172.18.0.1:50010, datanodeUuid=3abaf40c-4ce6-47e7-be2b-fbb4a7eba0e3, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-60401abd-4793-4acf-94dc-e8db02b27d59;nsid=1824008146;c=0) storage 3abaf40c-4ce6-47e7-be2b-fbb4a7eba0e3
... INFO blockmanagement.DatanodeDescriptor: Number of failed storage changes from 0 to 0
... INFO net.NetworkTopology: Adding a new node: /default-rack/172.18.0.1:50010
... INFO blockmanagement.DatanodeDescriptor: Number of failed storage changes from 0 to 0
... INFO blockmanagement.DatanodeDescriptor: Adding new storage ID DS-4ba1a710-a4ca-4cad-8222-cc5f16c213fb for DN 172.18.0.1:50010
... INFO BlockStateChange: BLOCK* processReport: from storage DS-4ba1a710-a4ca-4cad-8222-cc5f16c213fb node DatanodeRegistration(172.18.0.1:50010, datanodeUuid=3abaf40c-4ce6-47e7-be2b-fbb4a7eba0e3, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-60401abd-4793-4acf-94dc-e8db02b27d59;nsid=1824008146;c=0), blocks: 1, hasStaleStorage: false, processing time: 3 msecs
tcpdump ( Docker, , docker run --net=host), , (br-b59d498905c5 - Docker hadoop-network):
tcpdump -nnvvXS -s0 -i br-b59d498905c5 \
"(src host 172.18.0.3 or src host 172.18.0.2) and \
(dst host 172.18.0.3 or dst host 172.18.0.2)"
, IP- registerDatanode:
...
172.18.0.3.33987 > 172.18.0.2.9000: ...
...
0x0050: f828 004d 0a10 7265 6769 7374 6572 4461 .(.M..registerDa
0x0060: 7461 6e6f 6465 1237 6f72 672e 6170 6163 tanode.7org.apac
0x0070: 6865 2e68 6164 6f6f 702e 6864 6673 2e73 he.hadoop.hdfs.s
0x0080: 6572 7665 722e 7072 6f74 6f63 6f6c 2e44 erver.protocol.D
0x0090: 6174 616e 6f64 6550 726f 746f 636f 6c18 atanodeProtocol.
0x00a0: 01a7 010a a401 0a51 0a0a 3137 322e 3138 .......Q..172.18
0x00b0: 2e30 2e33 120f 6864 6673 2d64 6174 616e .0.3..hdfs-datan
0x00c0: 6f64 652d 311a 2433 6162 6166 3430 632d ode-1.$3abaf40c-
...
. , sendHeartbeat :
...
172.18.0.3.33987 > 172.18.0.2.9000: ...
...
0x0050: f828 004a 0a0d 7365 6e64 4865 6172 7462 .(.J..sendHeartb
0x0060: 6561 7412 376f 7267 2e61 7061 6368 652e eat.7org.apache.
0x0070: 6861 646f 6f70 2e68 6466 732e 7365 7276 hadoop.hdfs.serv
0x0080: 6572 2e70 726f 746f 636f 6c2e 4461 7461 er.protocol.Data
0x0090: 6e6f 6465 5072 6f74 6f63 6f6c 1801 9d02 nodeProtocol....
0x00a0: 0aa4 010a 510a 0a31 3732 2e31 382e 302e ....Q..172.18.0.
0x00b0: 3112 0f68 6466 732d 6461 7461 6e6f 6465 1..hdfs-datanode
0x00c0: 2d31 1a24 3361 6261 6634 3063 2d34 6365 -1.$3abaf40c-4ce
...
datanode , datanode BPServiceActor.register() , NameNode:
bpRegistration = bpNamenode.registerDatanode(bpRegistration);
namenode , IP- datanode .
, Docker:
import java.net.ServerSocket;
import java.net.Socket;
public class Server {
public static void main(String[] args) throws Exception {
ServerSocket server = new ServerSocket(9000);
Socket socket = server.accept();
System.out.println(socket.getInetAddress().getHostAddress());
}
}
import java.net.Socket;
public class Client {
public static void main(String[] args) throws Exception {
Socket socket = new Socket("172.18.0.2", 9000);
}
}
Server Client 172.18.0.2 172.18.0.2, Client, 172.18.0.3, 172.18.0.1.
( bridge network/docker0 9000) .
dfs.namenode.datanode.registration.ip-hostname-check, false namenode hdfs-site.xml, DNS-. , DNS, , IP-, datanodes, , DNS .
, registerDatanode, sendHeartbeat blockReport RegisterDatanodeRequestProto, HeartbeatRequestProto BlockReportRequestProto . DatanodeRegistrationProto . :
message DatanodeIDProto {
required string ipAddr = 1;
required string hostName = 2;
...
}