All geek questions in one place

Nadenode Hadoop Disk Size

Are there any suggestions regarding the size of the hard drive on the physical namenode? Of course, it does not store data from HDFS, for example, datanode, but what should I rely on when creating a cluster?

+4
source share
3 answers

The physical disk space in the NameNode really does not matter unless you run Datanode on the same node. However, it is very important to have a memory space (RAM) allocated for NameNode. This is because NameNode stores all HDFS metadata (block allocations, locks, etc.) in memory. If enough memory is not allocated, the NameNode may have run out of memory and crash.

+4
source

You may need some space to actually store the NameNode FSImage file, the edit file, and other related files.

In fact, it is recommended that you configure NameNode to use multiple directories (one local and another NFS mount) to save multiple copies of file system metadata. Thus, while the directories are on separate disks, a failure of one disk will not damage the metadata.

For details, see. This link .

+4
source

We hear from Cloudera that they recommend that nodes with names have faster drives — a combination of SSDs and 10kRPM SAS drives compared to typical SAT 2TB, 7200K drives. Whether this is reasonable or excessive, since everything I read suggests that you really don't need expensive high-speed storage for Hadoop.

-one
source

Source: https://habr.com/ru/post/1495959/

More articles:


All Articles