Native-lzo library is not available on Hadop datanodes

Question

I wrote a simple LzoWordCount for my gateway / hadoop -env.sh:

HADOOP_CLASSPATH=/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar JAVA_LIBRARY_PATH=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.105/lib/hadoop/lib/native/ 

When I run the MR job, I get:

 mapred.JobClient: Task Id : attempt_201307311800_0020_m_000002_2, Status : FAILED java.lang.RuntimeException: native-lzo library not available 

Any ideas on how to fix this? Did I notice that the grep native 'path to the hadop class returns nothing?

Answer 1

The problem turned out to be that we did not have lzop installed on datanodes. I fixed it using:

 sudo apt-get install lzop 

Answer 2

goto: https://code.google.com/p/hadoop-gpl-packing/downloads/list

load the jar and rpm install rpm (at that time: hadoop-gpl-packaging-0.6.1-1.x86_64.rpm ) and then

 yum install install lzop rpm -i hadoop-gpl-packaging-0.6.1-1.x86_64.rpm sudo cp /opt/hadoopgpl/native/Linux-amd64-64/* /usr/lib/hadoop/lib/native/ sudo cp /opt/hadoopgpl/lib/*lzo* /usr/lib/hadoop/lib/ 

be tired with the commands above so as not to override files you don't want. so first do the back of the target folder before starting it.

Answer 3

1.install lzo

 yum install lzo -y 

2.install hadoop-gpl-package

 wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/hadoop-gpl-packing/hadoop-gpl-packaging-0.6.1-1.x86_64.rpm rpm -i hadoop-gpl-packaging-0.6.1-1.x86_64.rpm 

3.copy jar

 cp /opt/hadoopgpl/lib/hadoop-lzo-*.jar $HADOOP_HOME/share/hadoop/common/ 

4 copy lib

 cp /opt/hadoopgpl/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/ 

Answer 4

So your hadoop-lzo-***.jar not in your HADOOP_CLASSPATH

Then you can copy your jar to your hadoop/lib folder or add the hadoop/bin/hadoop-env.sh path to it, you must do this on ALL of your sites

I hope he will solve it; -)