How to change yarn scheduler configuration in aws EMR?

Question

Unlike HortonWorks or Cloudera, AWS EMR does not seem to provide any graphical interface for changing the xml configurations of the various hadoop ecosystem infrastructures.

Logging into my namenode EMR and doing a quick

find \ -iname yarn-site.xml

I managed to find it in /etc/hadoop/conf.empty/yarn-site.xmland the performance planner to be located at /etc/hadoop/conf.empty/capacity-scheduler.xml.

But pay attention to how they are under conf.empty, and I suspect that these may not be the actual locations for the hml site and the xmls bandwidth scheduler.

I understand that I can change these configurations when creating a cluster, but I need to know how to change them without breaking the cluster.

I just want to play planning properties, etc. and try different schedulers to determine what can work with my spark applications.

Thanks in advance!

Answer 1

Well, yarn-site.xmland capacity-scheduler.xmlreally are in the correct locations ( /etc/hadoop/conf.empty/) and on the running cluster, edit them on the head node and restart YARN RM Daemon change the scheduler.

When promoting a new cluster, you can use EMR Configurations APIto change the corresponding values. http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html

: capacity-scheduler and yarn-site classifications EMR, XML.

: 4 2019 : Amazon EMR 5.21.0 . Amazon EMR, AWS (AWS CLI) AWS SDK.

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps-running-cluster.html