S3 rotation using log4j with elastic bean stack

Question

I am trying to migrate each log file to s3. There is an automatic script that raises tail_catina.log and overwrites anything there.

I need every log file: tail_catalina.log1 , tail_catalina.log2 , tail_catalina.log3 , etc., stored on my s3.

I want it all!

Answer 1

I use the basic combination of logrotate , s3cmd and cron to achieve this quite simply.

I made a detailed entry and explanation on my blog . It should work for anyone running Apache on a linux environment. I hope people find this useful, as it took me several hours to get the parts that I pulled out.

The following is a basic script, see blog post for line breaks:

 # rotate the logs! # common settings compress compresscmd /bin/gzip compressoptions -9 compressext .gz dateext dateformat -%Y-%m-%d-%s rotate 3 nomail missingok daily size 5k create 640 username username /var/logs/www.runpartner.com/*.log { sharedscripts postrotate sudo /usr/sbin/apache2ctl graceful /usr/bin/s3cmd sync /var/logs/www.runpartner.com/*.gz s3://bucket-logs/www.runpartner.com/ endscript } 

Answer 2

Early Beanstalk AMPs did not rotate magazines properly. You can fix this using the latest AMI in your deployment. Go to EC2 Console, AMI. Filter the list by selecting Amazon Images, "elasticbeanstalk", then sort by "Source" to see the latest AMIs.

Alternatively, you can edit the /etc/logrotate.conf.elasticbeanstalk file on the Beanstalk server to fix log rotation. The following configuration adds a timestamp after the file names. It produces logs such as tail_catalina.log-1322236861.gz, tail_catalina.log-1322240461.gz, etc.

 /var/log/tomcat6/catalina.out /var/log/tomcat6/monitor_catalina.log /var/log/tomcat6/tail_catalina.log { size 1M missingok rotate 2 compress notifempty copytruncate dateext dateformat -%s lastaction /bin/chown tomcat:elasticbeanstalk /var/log/tomcat6/*gz; /bin/chmod 664 /var/log/tomcat6/*gz endscript }