We have a sparking application running on HDFS 2.7.3 with yarn as a resource manager. When the application starts, these two folders:
/tmp/hadoop/data/nm-local-dir/filecache
/tmp/hadoop/data/nm-local-dir/filecache
are filled and therefore the disk. Therefore, from my research it turned out that setting these two properties in the yarn-site.xml file would help.
<property>
<name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.nodemanager.localizer.cache.target-size-mb</name>
<value>2048</value>
</property>
I configured them on yarn-site.xml for each namenode and masternode and restarted the yarn on masternode using stop-yarn.sh start-yarn.sh , however this does not seem to help. Am I missing something while making configuration changes? And how can we make sure that these changes apply to all work nodes?