All geek questions in one place

How to remove a node descriptor from DFS, but not from Mapred?

I am new to hadoop. To run some benchmarks, I need to distinguish the hadoop configuration for comparison.

I want to know the method for deleting a slave from DFS (not starting the datanode daemon), but not from Mapred (continue Tasktracker) or vice versa. AFAIK, there is one slave file for such chaos nodes, and not separate subordinate files for DFS and Mapred.

I am currently trying to run both DFS and Mapred on a slave node, and then kill the datanode on the slave. But it takes some time to migrate this node to dead nodes on the HDFS GUI. Any parameter can be configured to speed up this timeout?

Thankssss

+4
source share
2 answers

Try using dfs.hosts and dfs.hosts.exclude in hdfs-site.xml, mapred.hosts and mapred.hosts.exclude in mapred-site.xml. They are designed to allow / exclude hosts for connecting to NameNode and JobTracker.

After the list of nodes in the files has been updated accordingly, NameNode and JobTracker must be updated using the hadoop dfsadmin -refreshNodes and hadoop mradmin -refreshNodes respectively.

+6
source

Instead of using the slaves file to start all the processes in your cluster, you can only run the required daemons on each computer if you have few nodes.

0
source

Source: https://habr.com/ru/post/1385455/

More articles:


All Articles