I tried to create a schema for the json file manually and tried to create a Hive table, and I get a
column name length of type 10888 exceeds the maximum allowable length of 2000 .
I suppose I need to change the metastore details, but I'm not sure where the configuration is in the azure Hdinsights.
Another way I tried was I got the schema from the spark frame, and I tried to create a table from the view, but still getting the same error.
these are the steps that I tried in sparks
val tne1 = sc.wholeTextFiles("wasb:path").map(x=>x._2)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
val tne2 = sqlContext.read.json(tne1)
tne2.createOrReplaceTempView("my_temp_table");
sqlContext.sql("create table s ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'hive.serialization.extend.nesting.levels'='true') as select * from my_temp_table")
I get an error at this step
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: InvalidObjectException (: 5448 2000, struct
rdd, , . , .