Shared storage between multiple AWS EC2 instances

Question

I face the problem of sharing storage between multiple EC2 instances. I will have to do difficult tasks, so I need a lot of examples to do this. For one thing, I have an EBS volume attached to one server instance. On the other hand, I have a working instance. I created the AMI of this working instance, and then created several instances of this AMI instance. All work on one VPC. Basically, the server instance sends jobs, and workers complete the task. I would like to save some log files when my workers start jobs, in the partition store something like:

worker_1 / logfile.log

worker_2 / logfile.log

What could be the best solution for this?

• I read that it is not possible to add the same volume of EBS to multiple instances.
• I looked at GlusterFS , but here is what I found:

“Before we implement the proof of concept with two servers in different access zones by replicating an EBS volume with ext4 file system, we will list the cases when GlusterFS should not be used: Serial files are written simultaneously from several servers, such as logs. A blocking system can lead to serious problems if you store the logs in GlusterFS. The ideal solution is to store them locally and then use S3 to archive them. If necessary, we can consolidate several server logs before or and after storing them on S3.

• And finally, I also checked the S3 bucket mounted with s3fs , but I found that this is also not a good option:

"You cannot partially update the file using s3fs, so changing one byte will reload the entire file." Then, if you want to make a little gradual change, then it is definitely not. You cannot use s3fs - S3 just does not work that way, you cannot step by step modify a file. "

Then what can be a good solution to my problems and allow my employees to write their log files to a shared repository?

Thank you for your help!

Romanzo

Answer 1

Thanks for answers. but finally I use NFS between instances and it works very well!

Answer 2

As described in this thread , and in some of the answers already provided, two common ways to achieve this goal were to use S3 or NFS to share data between instances.

On April 9, 2015, Amazon announced the Amazon Elastic File System (Amazon Elastic File System) (EFS) , which provides a much better solution to the problem you are trying to solve.

Answer 3

Did you think that every worker can write their logs to a local disk (possibly even in the ephemeral section), and then force each worker to upload his own log file to S3 after its completion?

This is somewhat similar to what happens when you use Elastic MapReduce to run some distributed tasks in a Hadoop cluster.

You will get high write throughput (since it writes to a local disk if you use an ephemeral partition), as well as high throughput for sending files to S3 (since you will have the bandwidth of many workers available).

Answer 4

Not quite sure of the context, but is it possible to write objects directly on mounted S3?