AWS Cloudwatch Monitor for S3

Question

Amazon Cloudwatch provides some very useful metrics for monitoring my EC2, load balancers, elastic databases and RDS, etc. and allows me to set alarms according to a number of criteria; but is there a way to configure it to monitor my S3? Or are there any other monitoring tools (other than just enabling logging) that will help me track the number of POST / GET requests and data volumes for my S3 resources? And to provide alarms for activity thresholds or increased data storage?

Answer 1

I also could not find a way to do this using CloudWatch. Since April 2012, Derek @AWS answered this question since it does not have S3 support in CloudWatch. https://forums.aws.amazon.com/message.jspa?messageID=338089

The only thing I could think of was to import S3 access logs into a log service (e.g. Splunk). Then create a custom cloud view label where you publish the data you analyze from the logs. But then you need to filter out the survey of access logs and ... And while you were on it, you could just create alarms in Splunk, and not in S3.

If your use case is just to alert when you use it too much, you can set up a billing alert to use S3.

Answer 2

I think this may depend on where you want to track access. That is, if you are trying to measure / observe the use of S3 objects due to external http / https requests, then Anthony's suggestion, if you allow S3 registration and then import into splunk (or redshift) for analysis, may work. You can also see billing status for requests every day.

If you are trying to use use in your own applications, there are some AWS SDK metrics for cloud computing:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/metrics/package-summary.html

and

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/metrics/S3ServiceMetric.html

Answer 3

AWS S3 is a managed repository. The only metrics available in AWS CloudWatch for S3 are NumberOfObjects and BucketSizeBytes . To better understand your use of S3, you need to do extra work.

I recently wrote an AWS Lambda function to accomplish exactly what you are asking for, and it is available here:

https://github.com/maginetv/s3logs-cloudwatch

It works by analyzing server-side S3 log files and aggregates / exports metrics to AWS Cloudwatch (CloudWatch allows you to publish custom metrics).

Examples of graphs that you will receive in AWS CloudWatch after deploying this feature to your AWS account:

 RestGetObject_RequestCount RestPutObject_RequestCount RestHeadObject_RequestCount BatchDeleteObject_RequestCount RestPostMultiObjectDelete_RequestCount RestGetObject_HTTP_2XX_RequestCount RestGetObject_HTTP_4XX_RequestCount RestGetObject_HTTP_5XX_RequestCount + many others 

Since metrics are exported to CloudWatch, you can also easily configure alarms for them. The CloudFormation template is included in the GitHub repository, and you can quickly deploy this feature to gain visibility in using your S3 bucket.

EDIT 2016-12-10:

In November 2016, AWS added additional S3 request parameters to CloudWatch, which can be enabled when necessary. This includes metrics such as AllRequests , GetRequests , PutRequests , DeleteRequests , HeadRequests , etc. See Monitoring Performance with Amazon CloudWatch for more on this feature.

Answer 4

S3 is a managed service, which means that you do not need to take actions based on system events in order to support their work (while you can afford to pay for using the service). The spirit of CloudWatch is to help monitoring services that require you to take action to work.

For example, EC2 instances (which you manage yourself) typically need monitoring to prevent when they are overloaded or when they are underutilized or when they fly out; at some point, you need to take action to deploy new instances to scale, unscrew unused instances, to scale again or reload instances that have crashed. CloudWatch is designed to help you manage these resources more efficiently.