All geek questions in one place

AWS CLI S3 Copy without any errors

I am trying to upload a large CSV file (100 GB) from S3 to a mounted drive on an EC2 instance.

I downloaded AWS-CLI and used the command: aws s3 cp s3://bucket/file.csv . --region=us-east-1and everything seems to work fine and download about 70 GB of data, and then all of a sudden it simply ceases to exist without any messages or errors, deleting part of the created file.

Completed 642 of 916 part(s) with 1 file(s) remaining

and then he just stops.

Why is this possible and how can I get around this? I tried using s3cmd get, but it takes too much time and becomes very slow to such an extent that it almost stops loading anything.

Any ideas? Is there a problem with the network? disk problem? Problem with the API?

thanks

+4
source share
1 answer

Alternatively, you can try https://github.com/minio/mc , which implements both renewable downloads and downloads.

In addition, it also provides sessions for your current work, in case of any errors, it gracefully saves the session, and you can continue from where it left off.

mcwritten in the Golang and released under Apache license v2. Please give us feedback.

+5
source

Source: https://habr.com/ru/post/1616768/

More articles:


All Articles