How to change the Java version used by CDH

Question

I currently have CDH 5 installed on CentOS 6.5 with java jdk1.7, and I'm trying to get CDH to use jdk1.8.

I really know that Java 1.8 is not the recommended version of CDH, but it is just a test cluster, so this is not a big problem.

I installed both Java 1.7 and Java 1.8 from the Oracle site using the RPM installation, so both versions of Java are currently under /usr/java . Using ls -ld , my Java directory looks like this:

 /usr/java/default -> /usr/java/latest /usr/java/jdk1.7.0_75 /usr/java/jdk1.8.0_31 /usr/java/latest -> /usr/java/jdk1.8.0_31 

I also have a script installed in /etc/profile.d to set $JAVA_HOME to /usr/java/default . The contents of my profile .d script:

 export JAVA_HOME=/usr/java/default export PATH=${JAVA_HOME}/bin:${PATH} 

So, when I felt that I had this right, I run:

 $ which java /usr/java/default/bin/java 

Tell me that it points to the default version of Java. And to determine which version of java is running, I run:

 $ java -version java version "1.8.0_31" Java(TM) SE Runtime Environment (build 1.8.0_31-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode) 

And I see that Java 1.8 is currently running.

Everything seems great, except when I try to start the Hadoop service. The easiest launch is ZooKeeper, because it has only one service. HDFS has several servers, so work harder to just start and stop ZooKeeper.

ZooKeeper is launched using the following command:

 $ sudo service zookeeper-server start 

Then, to check which version of java is running, I look at the list of running processes for java :

 $ ps -ef | grep java 495 7170 1 7 12:27 ? 00:00:00 /usr/java/jdk1.7.0_75/bin/java -Dzookeeper.datadir.autocreate=false -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE -cp /usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.3.0.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/* -Dzookeeper.log.threshold=INFO -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg 

I know that he is running from the screen. The important part is that Zookeeper launches /usr/java/jdk1.7.0_75/bin/java .

To fix this, I tried several things:

• See the conf files for Hadoop and ZooKeeper in /etc/hadoop/conf and /etc/zookeeper/conf , respectively.

  I did not find anything by setting JAVA_HOME .

• /usr/bin/zookeeper looking at the /usr/bin/zookeeper script to find out if JAVA_HOME installed elsewhere,

  I found that script /usr/lib/bigtop-utils/bigtop-detect-javahome has the ability to set JAVA_HOME , but my profile .d script overrides this.

• Manually moving /usr/java/jdk1.7 to /tmp .

  Unfortunately, this is the only thing that works. When I move the jdk1.7 directory to another directory and start ZooKeeper, it will use Java 1.8. Moving jdk1.7 vertices back to ZooKeeper using Java 1.7.

Has anyone dealt with this problem and does anyone know how to deal with this? I feel like I configured Java correctly, but something tells ZooKeeper / Hadoop to use the old version of Java?

Answer 1

I hate to answer my questions, but here is the answer:

The incorrect version of $JAVA_HOME installed for two reasons:

• Using the service command removes most environment variables. From man service :

   service runs a System V init script in as predictable environment as possible, removing most environment variables and with current work- ing directory set to /. 

• /usr/lib/bigtop-utils/bigtop-detect-javahome script has the ability to be configured using the BIGTOP_JAVA_MAJOR environment variable to manually set which version of Java to use. I tried to set this as an environment variable, but service removes it :( It is also important to note that this script will find all versions of java, but the order of preferences is Java 6, Java 7, Java 8, Open Java. Therefore, if you have installed Java 6 and 8, she will prefer 6 over 8.

In short, to fix my problem, I added the following to the beginning of /usr/lib/bigtop-utils/bigtop-detect-javahome :

 BIGTOP_JAVA_MAJOR=8 

You can also set JAVA_HOME in this file to specify a specific version or path.

Answer 2

I came here because I was looking for ways to upgrade the JDK from 1.7 to 1.8 on the latest Coudera QuickStart VM 5.8 (I can't believe they still ship it with JDK1.7 by default!). The recommendations and suggestions in the above answers helped tremendously - but since they did not list the complete steps to achieve the update, I thought I would add that I would help others like me.

So, here is the complete set of steps to upgrade your Cloudera QuickStart VM from JDK1.7 to 1.8:

• check the current version of the JDK - out of the box:

   [ cloudera@quickstart ~]$ java -version java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) 

• download the desired version of JDK1.8.xx - in my case: jdk-8u111-linux-x64.tar.gz

as user 'cloudera':

• untar and move the resulting jdk1.8.0_111 directory to the / usr / java directory:

   tar xzf jdk-8u111-linux-x64.tar.gz sudo mv -f jdk1.8.0_111 /usr/java 

• disable all Hadoop services:

   $ for x in `cd /etc/init.d ; ls hadoop-*` ; do sudo service $x stop ; done 

• update bigtop-utils file - install JAVA_HOME in the new JDK:

    sudo vi /etc/default/bigtop-utils updated lines: # Override JAVA_HOME detection for all bigtop packages export JAVA_HOME=/usr/java/jdk1.8.0_111 

• update user cloudera.bash_profile - export JAVA_HOME and add PATH update:

   export JAVA_HOME=/usr/java/jdk1.8.0_111 PATH=$JAVA_HOME/bin:$PATH:$HOME/bin export PATH 

• reboot virtual machine

• check Java version - should now be 1.8:

   [ cloudera@quickstart ~]$ java -version java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode) 

By the way, I did not install / usr / java / default with the "last" symbolic link since @ milk3422 did it for the sake of simplicity, but it would work just as well.

thanks!

Answer 3

I had the same problem. I have installed

 JAVA_HOME=... below the file /usr/lib/bigtop-utils/bigtop-detect-javahome 

to override the default discovery value. It works great!

Answer 4

Java is configured in the web interface. Open Cloudera Manger → Hosts → Configurations → Advanced, then install JAVA HOME. This overrides the Java CDH discovery mechanism.