How would you read S3 as a hierarchical directory structure in Ruby?

Question

Has anyone read S3 buckets as subfolders?

folder1

- subfolder2

---- file3

---- file4

- file1

- file2

folder2

- subfolder3

- file5

- file6

My task is to read the folder1. I expect to see subfolder2, file1 and file2, but NOT file3 or file4. Right now, since I'm restricting the bucket keys to the prefix => 'folder1 /', you still get files 3 and 4, since they technically have the folder1 prefix.

It seems the only way to do this is to suck all the keys in folder1, and then use a string search to actually exclude files file3 and file4 from the result array.

Does anyone have any experience? I know that S3 FTP clients such as Transmit and Cyberduck should do this, but this is not obvious from the S3 interface itself.

Thanks Conrad

I looked at both AWS :: S3 and the right ones.

Answer 1

There is no folder concept in the S3 API. However, it allows file names with "/" in them, and it allows you to query with a prefix. You seem to already be familiar with this, but just wanted to be clear.

When you request with the prefix folder1/ , S3 will return everything under this "folder". To manipulate only direct descendants, you will have to filter the results yourself in Ruby (take your poison: reject or select ). This does not help improve performance (a common reason for using “folders” in S3), but it does its job.

Answer 2

You have encountered a limitation of the S3 API, and the only way to do this is to do filtering on the client.

The best (and most efficient) option is to “mirror” the S3 repository structure in the database file / xml, etc., and instead you ask against it. Then simply extract the files from S3 when the user has found the files they need.

Answer 3

Here is an example of using a virtual file system with an S3 driver.

As already mentioned, S3 does not have the concept of a folder, but provides the ability to fake it. The virtual file system uses these features to provide you with a "virtual folder"

http://alexeypetrushin.github.com/vfs/basics.html

http://alexeypetrushin.github.com/vfs/s3_basics.html

Answer 4

Update: For Version II AWS SDK

Amazon has now created iterators that allow the use of "prefix search." This can be used to emulate the directory / folder structure. In the above example (in PHP), the following should work:

 $client = S3Client::factory(array( 'key' => $this->aKey, 'secret' => $this->sKey, 'region' => $this->region, )); $iterator = $client->getIterator('ListObjects', array( 'Bucket' => 'folder1', 'Prefix' => 'subfolder2/', // supposing that the forward slash has been used to emulate diretcories )); foreach ($iterator as $object) { echo $object['Key'] . "\n"; // will echo only file 3 and file 4 }