S3cmd too many times

Question

I used to be a happy user s3cmd. However, recently, when I try to transfer a large zip file (~ 7Gig) to Amazon S3, I get this error:

$> s3cmd put thefile.tgz s3://thebucket/thefile.tgz .... 20480 of 7563176329 0% in 1s 14.97 kB/s failed WARNING: Upload failed: /thefile.tgz ([Errno 32] Broken pipe) WARNING: Retrying on lower speed (throttle=1.25) WARNING: Waiting 15 sec... thefile.tgz -> s3://thebucket/thefile.tgz [1 of 1] 8192 of 7563176329 0% in 1s 5.57 kB/s failed ERROR: Upload of 'thefile.tgz' failed too many times. Skipping that file. 

I am using the latest s3cmd on Ubuntu .

Why is that? and how can i solve it? If this is unsolvable, what alternative tool can I use?

Answer 1

In my case, the reason for the failure was the server time exceeding the S3 time. Since I used GMT + 4 on my server (located in the eastern US), and I used Amazon US East repository.

After setting up my server for Eastern time in the USA, the problem disappeared.

Answer 2

And now, in 2014, aws cli has the ability to upload large files instead of s3cmd.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html has installation / configuration instructions or often:

 $ wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip $ unzip awscli-bundle.zip $ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws $ aws configure 

followed by

 $ aws s3 cp local_file.tgz s3://thereoncewasans3bucket 

will give you satisfactory results.

Answer 3

I myself ran into this problem. I have a 24g.tar.gz file to host in S3.

Downloading smaller parts will help.

There is also a file size limit of ~ 5 GB, so I am breaking the file into pieces that can be reassembled when the fragments are downloaded later.

 split -b100m ../input-24GB-file.tar.gz input-24GB-file.tar.gz- 

The last part of this line is the "prefix". Split will add "aa", "ab", "ac", etc. To him. -b100m means 100 MB pieces. A 24-gigabyte file will contain about 240 parts of 100mb, called "input-24GB-file.tar.gz-aa" for "input-24GB-file.tar.gz-jf".

To merge them later, load them all into a directory and:

 cat input-24GB-file.tar.gz-* > input-24GB-file.tar.gz 

Taking md5sums of the source and split files and storing it in an S3 bucket, or, better, if it is not so big, using a system like parchive to be able to check, even fix some download problems, can also be useful.

Answer 4

I tried all the other answers, but no one worked. It looks like s3cmd is pretty sensitive. In my case, the s3 bucket was in the EU. Small files will load, but when it reaches ~ 60k, it always fails.

When I changed the ~ / .s3cfg file, it worked.

Here are the changes I made:

host_base = s3-eu-west-1.amazonaws.com

host_bucket =% (bucket) s.s3-eu-west-1.amazonaws.com

Answer 5

I had the same problem with ubuntu s3cmd.

 s3cmd --guess-mime-type --acl-public put test.zip s3://www.jaumebarcelo.info/teaching/lxs/test.zip test.zip -> s3://www.jaumebarcelo.info/teaching/lxs/test.zip [1 of 1] 13037568 of 14456364 90% in 730s 17.44 kB/s failed WARNING: Upload failed: /teaching/lxs/test.zip (timed out) WARNING: Retrying on lower speed (throttle=0.00) WARNING: Waiting 3 sec... test.zip -> s3://www.jaumebarcelo.info/teaching/lxs/test.zip [1 of 1] 2916352 of 14456364 20% in 182s 15.64 kB/s failed WARNING: Upload failed: /teaching/lxs/test.zip (timed out) WARNING: Retrying on lower speed (throttle=0.01) WARNING: Waiting 6 sec... 

The solution was to update s3cmd using the instructions from s3tools.org :

Debian and Ubuntu

Our DEB repository has been carefully created in the most compatible way - it should work for Debian 5 (Lenny), Debian 6 (Squeeze), Ubuntu 10.04 LTS (Lucid Lynx) and for all newer and possibly some older releases of Ubuntu. Follow these steps from the command line:

• Import S3tools Signature Key:

  wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add -

• Add repo to sources.list:

  sudo wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list

• Update the cache cache and install the latest s3cmd:

  sudo apt-get update && sudo apt-get install s3cmd

Answer 6

This error occurs when Amazon returns an error: they seem to disconnect the socket so that you do not load the gigabytes of the request in order to get the answer “no, this failed” in the response. This is why some people get it because of a skewed clock, some people get it because of policy errors, and others face size restrictions that require the use of a multi-part download API. It's not that everyone is mistaken or even looking at different problems: these are all different symptoms of the same basic behavior in s3cmd.

Since most error conditions will be deterministic, the behavior of s3cmd to discard the error message and repeat slowly seems crazy unsuccessful :(. Then. To get the actual error message, you can go to / usr / share / s 3cmd / S3 / S3.py (remembering that you need to delete the corresponding .pyc so that the changes are used) and add print e to the send_file except Exception, e: block.

In my case, I tried to set the Content-Type of the downloaded file to "application / x-debian-package". Apparently s3cmd S3.object_put 1) does not comply with the Content-Type passed through --add-header, but 2) cannot overwrite the Content-Type added through --add-header because it stores the headers in the dictionary with case - sensitive keys. The result is that it performs a signature calculation using the value of “content type” and then ends (with at least a lot of requests, this may be based on some kind of hashing order), sending the “Content-Type” to Amazon, which leads to a signature error.

In my particular case today, it seems that -M will cause s3cmd to guess the correct Content-Type, but it seems to be done based on the file name ... I would hope that it would use a mimemagic based database on the content file. Honestly, s3cmd cannot even return a failure status when it cannot load a file, so in combination with all these other problems, it's probably best to just write your own one-time tool to do the one thing you need ... it is almost certain that in the end it will save you time when you are bitten by some corner case of this tool: (.

Answer 7

s3cmd 1.0.0 does not yet support multithreading. I tried 1.1.0 beta and everything works fine. You can read about the new features here: http://s3tools.org/s3cmd-110b2-released

Answer 8

I had the same problem, it turned out to be a bad bucket_location value in ~/.s3cfg .

This blog post answered me.

If bucket youre uploading to does not exist (or you missed typing it), itll will fail. Thanks, general error message. - See more at: http://jeremyshapiro.com/blog/2011/02/errno-32-broken-pipe-in-s3cmd/#sthash.ZbGwj5Ex.dpuf

After checking my ~/.s3cfg we see that he had:

 bucket_location = Sydney 

Instead

 bucket_location = ap-southeast-2 

Correcting this value to use the proper name (s) solved the problem.

Answer 9

For me it worked:

In .s3cfg, I changed host_bucket

 host_bucket = %(bucket)s.s3-external-3.amazonaws.com 

Answer 10

s3cmd version 1.1.0-beta3 or higher will automatically use multipart uploads to allow sending arbitrarily large files ( source ). You can also control the size of the piece that it uses. eg.

 s3cmd --multipart-chunk-size-mb=1000 put hugefile.tar.gz s3://mybucket/dir/ 

This will download 1 GB of fragments.

Answer 11

I encountered the same error as the security group error, because the security policy was set incorrectly. I blame the S3 documentation.

I wrote about how to properly configure the policy on my blog, which:

 { "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads" ], "Resource": "arn:aws:s3:::example_bucket", "Condition": {} }, { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectVersion", "s3:GetObjectVersionAcl", "s3:PutObject", "s3:PutObjectAcl", "s3:PutObjectAclVersion" ], "Resource": "arn:aws:s3:::example_bucket/*", "Condition": {} } ] } 

Answer 12

In my case, I fixed this by simply adding rights.

 Bucket > Properties > Permissions "Authenticated Users" - List - Upload/Delete - Edit Permissions 

Answer 13

I encountered a similar error, which ultimately turned out to be caused by a temporary drift on the machine. Set the time correctly to fix the problem.

Answer 14

Locate the .s3cfg file, usually in your home folder.

If you have this, you will get a villain. Changing the following two options should help you.

 socket_timeout = 1000 multipart_chunk_size_mb = 15 

Answer 15

I turned to this simply without using s3cmd. Instead, I had great success with the python project, S3-Multipart on GitHub . It downloads and downloads, and also uses as many threads as needed.