Insert data into a static Hive partition using Spark SQL

Question

I find it difficult to figure out how to embed data in a static section of a Hive table using Spark SQL. I can use this code to write to dynamic sections:

df.write.partitionBy("key").insertInto("my_table")

However, I cannot figure out how to insert data into a static partition. This means that I want to define a section where the entire DataFrame should be written without the need to add a column to the DataFrame.

I see the static splitting mentioned in InsertIntoHiveTable , so I assume it is supported. Is there a public API to do what I want?

Answer 1

you can use

DataFrame tableMeta = sqlContext.sql(String.format("DESCRIBE FORMATTED %s", tableName));
String location = tableMeta.filter("result LIKE 'Location:%'").first().getString(0);

regex, . , ,

String partitionLocation = location + "/" + partitionKey

(partitionKey - - dt = 20160329/hr = 21)

df.write.parquet(partitionLocation)

( , dataframe, . , - )