Beehive: It is not possible to insert data into a table with 100 or more column columns. Error: in the column "PART_NAME", which has a maximum length of 767

Question

I work with a bush, I need to create a table with an "n" normal column and 100 or more as partition columns, and I can successfully create this table. Now when I come to load this table with the data of another table with the same schema, and all columns are non-split columns, I get the error as follows:

Failed to throw MetaException (message: attempt to save the value Error with MetaException exception (message: attempt to save the value "c1 = v1 / c2 = v2 / c3 = v3 / .... c100 = v100" in the column " PART_NAME ", which has the maximum length 767. Please correct your data!)

Considering the last line of the error, I tried to reduce the column name and their values, so that the resulting section path becomes shorter and it will work !! but this should not be the same as in real time the size of the script for the column name and their values ​​can be anything you like, or on the partition path.

eg. Here is my create query table:

CREATE TABLE xyz (c0 int) ALLOWED (c1 String, c2 String, c3 String, c4 String ....... c100 String) ROW FORMAT DELIMITED FIELDS TERMINATED by '|' REMEMBER HOW TO TEXT

And here is my insert in the request:

INSERT INTO TABLE xyz PARTITION (gc1, c2, c3 ...., c100) SELECT c0, c1, c2, c3, c4 ...., c100 FROM table123;

Am I doing something wrong or do I need to set some properties to use as many partitions as 100 or more?
Please give me the key, I'm stuck on this.
Thanks

Answer 1

I agreed with the experts that we should not go into so many sections in the table.

I would also like to quote this, since most nodes are based on unix / linux, and we cannot create a folder or file name longer than 255 bytes. This may be the reason for getting this error, because the section is only a folder.

Linux has a maximum file name of 255 characters in length for most file systems (including EXT4) and a maximum path of 4096 characters. eCryptfs is a multi-level file system.