All geek questions in one place

How to load data into BigQuery without ProtoBuf format error?

When loading data into BigQuery , I get the following error (copied from the job history in the BigQuery web console).

Errors: query: Failed to load FileDescriptorProto for '_GEN_DREMEL_ONESTORE_METADATA_SCHEMA_': (error code: invalidQuery) Field numbers 19000 through 19999 are reserved for the protocol buffer library implementation. Field numbers 19000 through 19999 are reserved for the protocol buffer library implementation. [... repeated a total of exactly 1000 times...] Field numbers 19000 through 19999 are reserved for the protocol buffer library implementation. (error code: invalidQuery)

Data is a managed back-up data warehouse. (The people on this team sent me to BigQuery for help.)

An error occurs with one of six randomly selected species; the rest are loaded successfully. In addition, when loading another type, the error "too many fields: 10693 (error code: invalid)" appears.

Both unsuccessful and successful views have a similar size of ~ 15 gigabytes of data.

What can we do to download this data?

+5
source share
1 answer

This is caused by the BigQuery restriction : a maximum of 10,000 columns per table. Therefore, the utility for downloading the Datastore backup just does not work in this case.

+3
source

Source: https://habr.com/ru/post/1257978/

More articles:


All Articles