All geek questions in one place

Copy data from AWS S3 to Aurora Postgres

I am trying to copy data from AWS S3 to Aurora Postgres. Currently my process is as follows:

  • Download files from S3 locally (to EC2 instance)
  • Run the command " COPY FROM STDIN ... " to load the data from the input stream into Aurora postrres.

I am wondering if there is a command to copy directly from S3 to Aurora postrres.

+7
source share
2 answers

AWS mentions this blog post with AWS Database Migration Service

https://aws.amazon.com/blogs/database/stream-data-into-an-aurora-postgresql-database-using-aws-dms-and-amazon-kinesis-data-firehose/

AWS DMS can read data from S3 source codes and load it into the target database. To do this, grant access to the S3 bucket containing one or more data files. In this S3 bucket, include a JSON file that describes the mapping between the data and the database tables in these files.

Source data files must be in comma-delimited (CSV) format. Name the files using the naming convention below. In this convention, schemaName is the source schema, and tableName is the name of the table in this schema.

+1
source

Copying data from S3 to Aurora PostgreSQL has been supported since version 11.1 using the aws_s3 extension.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.S3Import.html

0
source

Source: https://habr.com/ru/post/1266187/

More articles:


All Articles