All geek questions in one place

The main class was not found when compiling the Hadoop Sample Program

Hi, I am new to Hadoop and started studying a couple of days ago. I just followed the instructions from Digital Ocean to set up a Hadoop cluster . Subsequently, I just tried a simple sample program called WordCount from Hadoop docs .

My hadoop version is 2.5.1, which is the same version as in the tutorial, and runs on Ubuntu Precise. I guarantee that I made the correct setup, as stated in the tutorial. Here's the end of my content ~ .bashrc .

... #HADOOP VARIABLES START export JAVA_HOME=/usr/lib/jvm/java-7-oracle export HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin export PATH=$PATH:$HADOOP_INSTALL/sbin export HADOOP_MAPRED_HOME=$HADOOP_INSTALL export HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL export YARN_HOME=$HADOOP_INSTALL export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export HADOOP_PREFIX=/usr/local/hadoop #HADOOP VARIABLES END

Also, I checked my java home configuration and the result looks below

 sudo update-alternatives --config java There are 3 choices for the alternative java (providing /usr/bin/java). Selection Path Priority Status ------------------------------------------------------------ 0 /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1061 auto mode 1 /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1061 manual mode 2 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1051 manual mode * 3 /usr/lib/jvm/java-7-oracle/jre/bin/java 1 manual mode

So, I changed all the JAVA_HOME values ​​in the bashrc and hadoop-env.sh files to /usr/lib/jvm/java-7-oracle . I also guarantee that Dfs and Yarn start.

However, when I compile WordCount.java using this command

 hadoop com.sun.tools.javac.Main WordCount.java

Nothing goes my way. I got this error. Note that I use the Hadoop command instead of bin / hadoop, as the command works correctly, as it was defined in the bashrc file.

 Error: Could not find or load main class com.sun.tools.javac.Main

What is the possible cause of this error and how to get rid of it? I think this may be a Javapathpath problem, but I still can not figure out the details. All workarounds for this problem relate to running a java or javac command, not hadoop .

I just want to run the sample program first before starting to learn how it works. Any help would be greatly appreciated. Thanks..

+5
source share
7 answers

Try running from the hadoop directory

cd YARN_HOME bin / hadoop jar absolute path to jar file WordCount input path output path in hdfs

Check out the link below http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0

+2
source

I think Java is not enabled correctly. So go to hadoop-env.sh and enable java. Also check the java version and jre version. Both must have the same version.

+2
source

The Apache Hadoop tutorial assumes that environment variables are set as follows:

 export JAVA_HOME=/usr/java/default export PATH=$JAVA_HOME/bin:$PATH export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar

Perhaps the Digital Ocean Tutorial, which I also followed, should recommend adding these last two variables to ~ / .bashrc so that it looks like this:

 #HADOOP VARIABLES START export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export PATH=$JAVA_HOME/bin:$PATH export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar export HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin export PATH=$PATH:$HADOOP_INSTALL/sbin export HADOOP_MAPRED_HOME=$HADOOP_INSTALL export HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL export YARN_HOME=$HADOOP_INSTALL export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" #HADOOP VARIABLES END

It worked for my installation. See New compiled class files listed in the output:

Before:

 ubuntu@mail :/usr/local/hadoop$ ls bin include lib LICENSE.txt NOTICE.txt README.txt share WordCount.java etc input libexec logs output sbin WordCount_classes

After:

 ubuntu@mail :/usr/local/hadoop$ bin/hadoop com.sun.tools.javac.Main WordCount.java ubuntu@mail :/usr/local/hadoop$ ls bin input LICENSE.txt output share WordCount$IntSumReducer.class etc lib logs README.txt WordCount.class WordCount.java include libexec NOTICE.txt sbin WordCount_classes WordCount$TokenizerMapper.class

Another useful resource was the following:

http://ubuntuforums.org/archive/index.php/t-634996.html Add the following lines to the open .bashrc file, save it and close: export JAVA_HOME = "/ usr / lib / jvm / java-6-sun- 1.6.0.22 "export PATH = $ PATH: $ JAVA_HOME / bin issues the following command in the terminal: source $ HOME / .bashrc

Please refer to this blog post for more information ( http://sureshatt.blogspot.com/2011/01/easiest-way-to-install-java-in-ubuntu.html )

+2
source

Assuming you are using Eclipse or any other IDE ,
As I mentioned in this , create a simple maven based wordcount project with your class.
Thus, all dependencies will be processed. Then Right click in your project and select the Export option and specify the name <hadoop-example>.jar , and then the following, then create a jar file for your wordcount project.

You do not need to explicitly compile your programs, Eclipse will do this for you, once the export is successful

If you installed hadoop on the same computer, then start all the daemons and check with jps whether all the daemons are running or not.
Else copy the jar file to the virtual machine where hasoop is installed. Go to the location of the can and run the following command.

 hadoop jar <hadoop-example>.jar <fully qualified main class name> hdfsInputPath hdfsOutputPath

This will launch your main class ( WordCount in your case)
You need to use the above command to start any hadoop program using the command line using the jar file, and then the main class name ( WorkCount ) in your case.
You can make WordCount as the main entry point to the application when exporting the jar file. Therefore, you do not need to give a fully qualified name. that would be so.

 hadoop jar <hadoop-example>.jar hdfsInputPath hdfsOutputPath

Please try this and let us know if this helps you.
Update: As mentioned in the comments, the IDE is not used.

The above paths are used by Map/Reduce when starting the program.

But before you start, you have to make a jar that needs to compile all the dependents.
Therefore, take one variable as HADOOP_CLASSPATH_LIBS and assign all the banks in the file <installed-hadoop>/share/hadoop/<subfolder>/lib .
For every jar, you should give the absolute path and export this variable .
This will allow you to compile and then create a jar file. After you use the jar, you can follow the steps above to start it. If you need more help, you can help. Hope this helps.

+1
source

Hadoop requires a jdk path for JAVA_HOME. Make sure you set the jdk path, not the jre. It seems you installed java manually. Check the javac version to make sure it is enabled.

 javac -version

Check out a similar answer .

+1
source

Try setting the environment variable HADOOP_CLASSPATH

 export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar:<path to hadoop libs>
0
source

Hadoop works with both openjdk and oracle jdk, but you are using oracle jdk. I had the same problem, so I did the following.

 1)export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 2)export PATH=${JAVA_HOME}/bin:${PATH} 3)export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

After running this command in the terminal, you can compile the java file. Hadoop cannot find the correct Java path, so you get this error.

0
source

Source: https://habr.com/ru/post/1206384/

More articles:


All Articles