Manage large log files

Question

We have a crawler that aggressively scans our target sites, so the log files look pretty huge. In some cases, more than 1 GB, I am not too convenient when deleting or overwriting. Any examples of how you managed large log files?

Answer 1

Use cron script to rotate log files daily. You will basically rename your logfile.logto logfile-YYYY-MM-DD.log. Thus, instead of one huge log file, you have smaller ones and you can easily find registered messages from a certain period of time. If you also compress your rotated logs, you will save even more disk space.

Answer 2

You can also use log4net, log4j, or other logging frameworks that automatically change file names based on file size or date. http://logging.apache.org/log4net/