All geek questions in one place

Cloudera Manager: Where can I put Java ClassPath jobs for MapReduce?

I have a Hadoop-Lzo that works happily on my local pseudo-cluster, but the second I try to use the same jar file in production, I get:

java.lang.RuntimeException: native-lzo library not available

Libraries are checked in DataNodes, so my question is:

In what screen / setup do I indicate the location of the native-lzo library?

+4
source share
2 answers

For MapReduce, you need to add entries to the MapReduce Client Environment Safety valve. You can find MapReduce Client Safety by going to the View and Edit tab in the Configuration section. Then add the following lines:

  • HADOOP_CLASSPATH = $ HADOOP_CLASSPATH: / opt / Cloudera / parcels / HADOOP_LZO / Library / Hadoop / Library / *
  • JAVA_LIBRARY_PATH = $ JAVA_LIBRARY_PATH: / opt / Cloudera / packages / HADOOP_LZO / Library / Hadoop / Library / native

Also add LZO codecs to the io.compression.codecs property in the MapReduce service. To do this, go to io.compression in the View and edit section in the Configuration section, and these lines:

  • com.hadoop.compression.lzo.LzoCodec
  • com.hadoop.compression.lzo.LzopCodec

Remember to restart your MR daemons after making the changes. After restarting, reconfigure the MP client.

Detailed help on using LZO can be found at this link: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Installation-Guide/cmig_install_LZO_Compression.html

NTN

+3
source

try sudo apt-get install lzop on your TaskTracker nodes.

0
source

Source: https://habr.com/ru/post/1495388/

More articles:


All Articles