YARN DRF Explanation

Question

I read the 4th Edition Hadoop The Definitive Guide and stumbled upon this explanation for YARN DRF (in Chapter 4, Dominant Resource Justice)

Imagine a cluster with a total of 100 processors and 10 TB of memory. Appendix A requests containers (2 processors, 300 GB), and application B requests containers (6 processors, 100 GB). Since the query (2%, 3%) of the cluster, so the memory dominates, because its share (3%) is greater than the processors (2%). Request Bs (6%, 1%), so the central processor is dominant. Since requests for Bs containers are twice as large in the dominant resource (6% versus 3%), he will be allocated half as many containers under fair placement.

I can not understand the meaning it will be allocated half as many containers under fair sharing. I think ithere Application B, but Application Bhalf the number of Application A containers stands out. Is that right? Why Application Ballocates smaller containers, even if it requires more resources?

Any suggestion and reference to any explanatory document would be appreciated so much. Thank you in advance.

Answer 1

The dominant resource calculator is based on the concept of dominant resource equity (DRF).

To understand DRF, you can refer to this article here: https://people.eecs.berkeley.edu/~alig/papers/drf.pdf

In this article, see section 4.1 for an example.

DRF ( A = B).

Total Resouces Available: 100 , 10000

Requirements of Application A: 2 , 300

Requirements of Application B: 6 , 100

A dominant resource is Memory (2% 3% )

B dominant resource is CPU (6% 1% )

, "A" x , "B" - y.

• A

  2x CPUs + 300x GB Memory (2 CPUs and 300 GB Memory for each container)
  

• B:

  6y CPUs + 100y GB Memory (6 CPUs and 100 GB Memory for each container)
  

• :

  2x + 6y <= 100 CPUs
  
  300x + 100y <= 10000 GB Memory
  

• DRF A B.

  A dominant need: 300x / 10000 GB (300x out of 10000 GB of total memory)
  
  B dominant need: 6y / 100 CPUs (6y out of 100 CPUs)
  
  DRF will try to equalise: (300x / 10000) = (6y / 100)
  
  Solving the above equation gives: x = 2y
  

x = 2y 3, x = 20 y = 10.

:

• Application A is allocated 20 containers: (40 CPUs, 6000 GB of Memory)
• Application B is allocated 10 containers: (60 CPUs, 1000 GB of memoty)

, :

Total allocated CPU is: 40 + 60 <= 100

Total allocated Memory is: 6000 + 1000 <= 10000

, :

Since B’s container requests are twice as big in the dominant resource (6% 
versus 3%), it will be allocated half as many containers under fair sharing.