When I ran the copy command to copy all the files from the S3 folder to the Redshift table, it does not work with : "ERROR: gzip: unexpected end of stream. Unknown zlib error code. zlib error code: -1"
copy table_name
(column_list)
from 's3://bucket_name/folder_name/'
credentials 'aws_access_key_id=xxxxxx;aws_secret_access_key=xxxxxxxxx'
delimiter '|' GZIP
However, when I specify the file prefix for each of the files in the folder, it succeeds:
copy table_name
(column_list)
from 's3://bucket_name/folder_name/file_prefix'
credentials 'aws_access_key_id=xxxxxx;aws_secret_access_key=xxxxxxxxx'
delimiter '|' GZIP
Files have gzip-ed.
The AWS document does not explicitly state that if you simply specify the folder_name, it will be fine if the copy command loads the entire contents of this folder, however I get an error message.
Has anyone encountered similar problems? Is a file prefix required for this kind of operation?
source
share