All geek questions in one place

How can I organize a million + files now when I upgrade to Amazon S3?

Well, I boot from my shared host, and I move on to a combination of VPS from Linode and Amazon S3 accommodates several million JPEGs.

My great concern is some sanity with all these images. Is there any hope for this? I understand that you are allowed to use only 100 "buckets" and "buckets" - the only type of structure in S3.

Does a few million files put something in the bucket that you would recommend?

+6
source share
1 answer

You can notice in the limitations and restrictions of the branches , it says:

There is no limit to the number of objects that can be stored in a bucket

My experience is that a very large number of objects in one bucket will not affect the performance of getting one object by its key (i.e. get seems to have constant complexity).

Having a very large number of objects does not affect the speed of listing a given number of objects :

 List performance is not substantially affected by the total number of keys in your bucket

However, I must warn you that most of the S3 management tools that I used (for example, S3Fox) will drown out and die from a terrible slow death when trying to access a bucket with a very large number of objects. One tool that seems to do well with a very large number of objects is the S3 Browser (they have a free version and a Pro version, I am not related to them in any way).

Using "folders" or prefixes does not change any of these points (receiving and listing a given number of objects is still constant, most of the tools still fall on themselves and freeze).

+5
source

Source: https://habr.com/ru/post/906023/

More articles:


All Articles