All geek questions in one place

YARN shell command to get the number of containers and vcores used when starting applications

hadoop job -listseems to only show mapreduce jobs, but my resource manager user interface shows things like “VCores Dedicated Processors” and “Allocated Memory MB” for all jobs running on YARN (including things like Spark and Tez).

How can I get these results through the command line instead of navigating to the user interface?

+4
source share
1 answer

YARN supports various types of applications. MapReduce is one of the types of applications supported by YARN. If you are using hadoop job(which is deprecated, you should use it instead mapred job) or mapred job, you can only manage MapReduce jobs.

To view the status of various types of applications (mapreduce, spark, etc.), you should use the YARN CLI.

For example, "application-thread -app -appStates ALL" displays the status of all applications. The result of this command contains the Application Type column , which indicates the type of application (for example, MAPREDUCE for a MapReduce application).

You can get information about the application, application attempt, containers, etc. The commands for version 2.7.1 of Hadoop are given here: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html .

But these commands do not return details, such as "VCores Dedicated CPUs" and "Allocated Memory MB", through the CLI.

Team

yarn application -status {Application ID}returns "Aggregate resource allocation" in terms of "MB-seconds" and "vcore-seconds"

For example, -statusfor one of my applications:

Cumulative allocation of resources: 12865641 MB-seconds, 1041 seconds vcore

+2
source

Source: https://habr.com/ru/post/1611966/

More articles:


All Articles