Merge multiple zip files on s3 into fewer zip files

Question

We have a problem when some of the files in the s3 directory are in the range of ~ 500MiB, but many other files are in KiB and Bytes. I want to combine all the small files into several large files of the order of ~ 500MiB.

What is the most efficient way to overwrite data in s3 folder instead of downloading, merging on local and back to s3. Is there any utility / aws command that I can use to achieve it?

Answer 1

S3 is a storage service and does not have the ability to compute. For what you ask, you need to calculate (combine). Thus, you cannot do what you want without loading, merging and unloading.