Hadoop: writing and reading HDFS files

Question

I have a basic question regarding writing and reading files in HDFS.

For example, if I write a file using the default configuration, Hadoop internally must write each block to 3 data nodes. I understand that for each block, the client first writes the block to the first node data in the pipeline, which will then inform the second, etc. After the third node data has successfully received the block, it returns a confirmation to the node 2 data and, finally, to the client through Data node 1. Only after receiving confirmation for the block is the record considered successful, and the client continues to the next block .

If so, then not the time taken to write each block is longer than the traditional file write because of -

• replication rate (default is 3) and
• the recording process is sequentially blocked after the block.

Please correct me if I am mistaken in my understanding. In addition, the following questions are below:

• My understanding is that reading / writing files in Hadoop does not have parallelism, and the best thing it can do is the same as for normal reading or writing a file (i.e. if replication is set to 1 ) + some overhead in the distributed communication mechanism.
• Parallelism is provided only at the stage of data processing through Map Reduce, but not during the reading / writing of the file by the client.

Answer 1

Although your description of the file entry above is correct, the DataNode can read and write data at the same time . From the HDFS Architecture Guide :

The DataNode can receive data from the previous one in the pipeline and at the same time forward data to the next in the pipeline

A write operation takes longer than in a traditional file system (due to bandwidth problems and general overhead), but not up to 3x (provided that the replication rate is 3).

Answer 2

I think your understanding is correct.

One would expect a simple HDFS client to write some data, and when at least one block replica has been recorded, it returns a control, while HDFS generates other replicas asynchronously.

But in Hadoop, HDFS is being developed around the “write once, read many times” template, so the focus was not on write performance.

On the other hand, you can find parallelism in the Hadoop MapReduce (which you can also see the HDFS client), designed to explain this.

Answer 3

HDFS write operation:

There are two options.

dfs.replication : Replication of blocks by default. The actual number of repetitions may be indicated when creating the file. By default, used if replication is not specified at creation time.

dfs.namenode.replication.min : minimal block replication.

Even if dfs.replication set to 3, the write operation will be considered successful after dfs.namenode.replication.min ( default value : 1 ) is replicated.

But this replication before dfs.replication will happen in the serial pipeline . The first Datanode writes the block and redirects it to the second Datanode. The second Datanode writes the block and passes it to the third Datanode.

DFSOutputStream also maintains an internal queue for packets awaiting acknowledgment by datanodes called the ack queue. A packet is removed from the ack queue only when it has been acknowledged by all Datanodes in the pipeline.

Take a look at the related SE question: Hadoop 2.0 data record confirmation confirmation

HDFS read operation:

HDFS read operations occur in parallel instead of sequential write operations