Spark 2.1 - Error creating instance of HiveSessionState

Question

On a new installation of Spark 2.1, I get an error when I execute the pyspark command.

Traceback (most recent call last):
File "/usr/local/spark/python/pyspark/shell.py", line 43, in <module>
spark = SparkSession.builder\
File "/usr/local/spark/python/pyspark/sql/session.py", line 179, in getOrCreate
session._jsparkSession.sessionState().conf().setConfString(key, value)
File "/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/local/spark/python/pyspark/sql/utils.py", line 79, in deco
raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"

I have Hadoop and Hive on the same machine. Hive is configured to use MySQL for the metastore. I did not get this error using Spark 2.0.2.

Can anyone point me in the right direction?

Answer 1

I had the same problem. Some of the answers sudo chmod -R 777 /tmp/hive/, or in order to lower the spark from hadoop to 2.6, did not work for me. I realized that the cause of this problem for me is that I was executing SQL queries using sqlContext, instead of using sparkSession.

sparkSession =SparkSession.builder.master("local[*]").appName("appName").config("spark.sql.warehouse.dir", "./spark-warehouse").getOrCreate()
sqlCtx.registerDataFrameAsTable(..)
df = sparkSession.sql("SELECT ...")

this works great for me now.

Answer 2

Windows, .

shell.py .enableHiveSupport()

 spark = SparkSession.builder\
            .enableHiveSupport()\
            .getOrCreate()

, :

spark = SparkSession.builder\
        .getOrCreate()

shell.py . "C:\spark-2.1.1-bin-hadoop2.7\python\pyspark"

,

Answer 3

Spark 2.1.0 - - , " " org.apache.spark.sql.hive.HiveSessionState ': ".

.

Answer 4

HADOOP_CONF_DIR. hadoop, pyspark , hadoop, .

, HADOOP_CONF_DIR, ,

.

Answer 5

.

, Scala 2.11, Spark 2.1, .

https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.11/2.1.0

Answer 6

I saw this error on the new (2018) Mac that came with Java 10. It was fixed to install JAVA_HOMEon Java 8:

export JAVA_HOME='usr/libexec/java_home -v 1.8'

Answer 7

I also fought in cluster mode. The hive-site.xml file from the sparkconf directory has been added, if you have an hdp cluster, it should be located at / usr / hdp / current / spark2-client / conf. This works for me.

Answer 8

I was getting this error while trying to start pyspark and spark-shell when my HDFS was not running.