AWS Datapipeline RedShiftCopyActivity - How to Specify Columns

Question

I am trying to copy a bunch of csv files from S3 to Redshift using RedShiftCopyActivity and datapipeline.

This works fine as long as the csv structure matches the table structure. In my case, csv has fewer columns than the table, and then RedShiftCopyActivity does not work with the error "Delimiter not found" in stl_load_errors.

I would like to use the "columns" command of the copy redshift command. That way I can make it work, but part of the redshift copy command columns is not available in RedShiftCopyActivity.

Are there any suggestions?

All tips are welcome.

Thank you very much in advance.

Peter

Answer 1

I know this is an old question, but now you can list the columns in the Redshift COPY command.

COPY tablename (column1 [,column2, ...]) 

When loading data from S3, the column order should correspond to the order of the source data. Check out the docs here: Amazon Redshift Column Mapping Options .

Radu