Measuring traffic from the Apache access log

Question

Can I measure how much traffic was used in a single Apache log file?

Format:

66.249.72.214 - - [05/Nov/2011:12:47:37 +0200] "GET /produktas/565638 HTTP/1.1" 200 4699 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 

As far as I understand, 4699 are bytes which were transferred except for headers.

I need a simple solution (maybe a little bash script) to sum the bytes in each line of the log.

Answer 1

Try it. I tested it in a local file, but I can not determine if it works in all configurations / locales / ...

 cat apache.log | perl -e 'my $sum=0; while(<>) { my ($traffic) = m/\[.+\] ".+" \d+ (\d+)/; $sum += $traffic}; print "$sum\n"' 

January 2017 Update:. In the meantime, I learned more about Perl and how to do it today:

 cat apache.log | perl -nE '/\[.+\] ".+" \d+ (\d+)/; $sum += $1; END {say $sum}' 

Answer 2

For detailed monitoring of the log file and actual bandwidth usage, go to AWStats .

It takes the Apache log file as input and gives you a very detailed analysis of visitors and bandwidth with graphs.

You can also try GoAccess .

Answer 3

Apache Access Log - Using Global Bandwidth:

 awk '{ s += $10 } END { print "Total ", s/1024/1024 " Mo", "- Moyenne ", s/NR/1024/1024 " Mo", "- Accès ", NR }' access.log 

And for the file:

 grep NAME_OF_RESOURCE_HERE /var/log/apache2/access.log* | awk '{ s += $10 } END { print "Total ", s/1024/1024 " Mo", "- Moyenne ", s/NR/1024/1024 " Mo", "- Accès ", NR }' 

You will get something like this: Total 301.985 Mo - Moyenne 0.0430055 Mo - Access 7022

Answer 4

I think you need to use apachetop utility, try installing the following command from APT:

 sudo apt-get install apachetop 

And then run it with the command:

 sudo apachetop -f /path/to/access.log 

And you are rock! :)

Answer 5

We needed to get traffic for the last X days. I'm really not in Perl, so I did this:

 zcat $(find -name yourvhost_access.log*.gz -mtime -3 2>/dev/null| xargs ) \ | awk '$10 ~ /^[0-9]+$/ {print $10}' \ | paste -sd+ \ | bc 

Steps:

1. find the last 3 accesslogs reached
2. print position 10 if it is a number - the payload should be here
3. put it along with a +
4. count