Yarn: .xml thread site does not take effect

We have a sparking application running on HDFS 2.7.3 with yarn as a resource manager ... when the application starts .. these two folders

/tmp/hadoop/data/nm-local-dir/filecache

/tmp/hadoop/data/nm-local-dir/filecache

are filled and therefore the drive .....

therefore, from my research it was found that setting these two properties in the yarn-site.xml file will help

<property>
        <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
        <value>2000</value>
</property>

<property>
         <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
         <value>2048</value>
</property>

I configured them on each namenode and masternode .. and restarted the yarn on masternode .. but it still doesn’t help ... please let me know if I missed something ... when setting up the change ... it's yarn- site.xml

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hdfs-name-node</value>
</property>

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>16384</value>
</property>

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>6</value>
</property>

<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>16384</value>
</property>

<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>file:///tmp/hadoop/data/nm-local-dir</value>
</property>

<property>
    <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
    <value>2000</value>
</property>

<property>
     <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
     <value>2048</value>
</property>

</configuration>

Update : I tried with this property, but still not clearing, and the disk becomes full

<property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value>
   </property>
+1
1

, "/tmp/hadoop/data/nm-local-dir/filecache", .

, , , "yarn.nodemanager.localizer.cache.cleanup.interval-ms" "yarn.nodemanager.localizer.cache.target-size-mb" "/tmp/hadoop/data/nm-local-dir/filecache", "yarn.nodemanager.local-dirs" file:///tmp/hadoop/data/nm-local-dir "filecache" . "file:///tmp/hadoop/data/nm-local-dir" ":///tmp/hadoop/data/nm-local-dir/filecache".

0

Source: https://habr.com/ru/post/1666545/


All Articles