Spark connection with Phoenix NoSuchMethod Exception

Question

I am trying to connect to Phoenix via Spark / Scala to read and write data as a DataFrame. I follow the example of GitHub , however, when I try the very first example of Download as DataFrame using the data source API, I get the following exception.

An exception in the "main" thread java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL (Z) Lorg / apache / hadoop / hbase / client / Put;

There are a few things that drive me crazy with these examples:

1) The import import org.apache.phoenix.spark._ statement import org.apache.phoenix.spark._ gives me the following exceptions in my code:

can't resolve phoenix character

I have included below jars in my sbt

 "org.apache.phoenix" % "phoenix-spark" % "4.4.0.2.4.3.0-227" % Provided, "org.apache.phoenix" % "phoenix-core" % "4.4.0.2.4.3.0-227" % Provided, 

2) I get an outdated warning to load a character.

I figured out this warning, but did not get any links, and I could not find a single example of the proposed method. I cannot find any other good resource that will help you connect to Phoenix. Thank you for your time.

Answer 1

use .read instead of loading as shown below.

 val df = sparkSession.sqlContext.read .format("org.apache.phoenix.spark") .option("zkUrl", "localhost:2181") .option("table", "TABLE1").load() 

Answer 2

It’s too late to answer, but here is what I did to solve a similar problem (different method not found and warning about failure):

1.) About NoSuchMethodError: I took all the jars from the hbase installation folder and add it to your project. Also add pheonix spark tanks. Be sure to use compatible versions of sparks and pheonix sparks. Compatible with 2.0+ with pheonix-spark-4.10 + maven-central-link . This resolved NoSuchMethodError

2.) About the download - The download method has long been outdated. Use sqlContext.phoenixTableAsDataFrame. For reference, see this Loading as a DataFrame Directly Using a Configuration Object