All geek questions in one place

Import flat data files into the hive without defining the hive table structure

Is it possible to import CSV or any other flat files into bushes without creating and defining the table structure first in the bush. Say my csv file has 200 columns and needs to be imported into the hive table. Therefore, I must first create a table in the hive and determine all the column names and data type in this catch table and import. Is there a way by which I can import directly into the bush and it automatically creates a table structure from the first row, say, similar to sqoop import?

+4
source share
2 answers

use sqoop with the "hive-import" switch and it will create your table http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_importing_data_into_hive for you

+3
source

Check your hive-site.xml for the value of the javax.jdo.option.ConnectionURL property. If you do not define this explicitly, the default will use the relative path to create the metastore hive (jdbc: derby :; databaseName = metastore_db; create = true), which will differ depending on where you are starting the process from. This explains why you do not see the table through the show tables.

To overcome this, you need to define this property value in your hive-site.xml using an absolute path

+2
source

Source: https://habr.com/ru/post/1432642/

More articles:


All Articles