How to organize a large number of objects in the cloud storage?

Question

I am looking for suggestions on how to organize a large number of objects.

Assuming an incoming rate of about 60,000,000 files per day, and I would like to keep them for 180 days.

With an hourly partition at the top level there will be 4320 (24 * 180) directories. And each directory will contain an average of ~ 2,500,000 files.

If I only need to get the files individually by its full path, and I do not need to indicate the contents of the directory, is there any problem leaving all 2500000 of them at the same level?

Or do I need to hash file names and store them in multiple subdirectories? (for example, as is usually done when saving to a traditional file system)

Answer 1

There is no limit to the number of objects that you can store in a bucket, and dividing objects into more "subdirectories" does not affect scalability or the difference in performance. For the Google Cloud Storage service, all object names are flat: the “/” in the path just looks like any other character in the object name.

Mike Schwartz, Google Cloud Storage Team