How would I calculate time indicators (hourly average) based on log file data?
Let me make this clearer, consider a log file that contains entries as follows: each UID is displayed only twice in the log. they will be implemented in xml format. And they are likely to appear out of sequence. And the log file will only have data for one day, so only one day will be available.
There are no UIDs in the log file.
I need to know the average hourly response time for these requests. The following is the request and response in the log file. UID is the key that binds the b / w request and response.
2013-04-03 08:54:19,451 INFO [Logger] <?xml version="1.0" encoding="UTF-8" standalone="yes"?><log-event><message-time>2013-04-03T08:54:19.448-04:00</message-time><caller>PCMC.common.manage.springUtil</caller><body><log-message-body><headers>&lt;FedDKPLoggingContext id="DKP_DumpDocumentProperties" type="context.generated.FedDKPLoggingContext"&gt;&lt;logFilter&gt;7&lt;/logFilter&gt;&lt;logSeverity&gt;255&lt;/logSeverity&gt;&lt;schemaType&gt;PCMC.MRP.DocumentMetaData&lt;/schemaType&gt;&lt;UID&gt;073104c-4e-4ce-bda-694344ee62&lt;/UID&gt;&lt;consumerSystemId&gt;JTR&lt;/consumerSystemId&gt;&lt;consumerLogin&gt;jbserviceid&lt;/consumerLogin&gt;&lt;logLocation&gt;Beginning of Service&lt;/logLocation&gt;&lt;/fedDKPLoggingContext&gt;</headers><payload> &lt;ratedDocument&gt; &lt;objectType&gt;OLB_BBrecords&lt;/objectType&gt; &lt;provider&gt;JET&lt;/provider&gt; &lt;metadata&gt;&amp;lt;BooleanQuery&amp;gt;&amp;lt;Clause occurs=&amp;quot;must&amp;quot;&amp;gt;&amp;lt;TermQuery fieldName=&amp;quot;RegistrationNumber&amp;quot;&amp;gt;44565153050735751&amp;lt;/TermQuery&amp;gt;&amp;lt;/Clause&amp;gt;&amp;lt;/BooleanQuery&amp;gt;&lt;/metadata&gt; &lt;/ratedDocument&gt; </payload></log-message-body></body></log-event> 2013-04-03 08:54:19,989 INFO [Logger] <?xml version="1.0" encoding="UTF-8" standalone="yes"?><log-event><message-time>2013-04-03T08:54:19.987-04:00</message-time><caller>PCMC.common.manage.springUtil</caller><body><log-message-body><headers>&lt;fedDKPLoggingContext id="DKP_DumpDocumentProperties" type="context.generated.FedDKPLoggingContext"&gt;&lt;logFilter&gt;7&lt;/logFilter&gt;&lt;logSeverity&gt;255&lt;/logSeverity&gt;&lt;schemaType&gt;PCMC.MRP.DocumentMetaData&lt;/schemaType&gt;&lt;UID&gt;073104c-4e-4ce-bda-694344ee62&lt;/UID&gt;&lt;consumerSystemId&gt;JTR&lt;/consumerSystemId&gt;&lt;consumerLogin&gt;jbserviceid&lt;/consumerLogin&gt;&lt;logLocation&gt;Successful Completion of Service&lt;/logLocation&gt;&lt;/fedDKPLoggingContext&gt;</headers><payload>0</payload></log-message-body></body></log-event>
here is the bash script I wrote.
uids=cat $i|grep "Service" |awk 'BEGIN {FS="lt;";RS ="gt;"} {print $2;}'| sort -u for uid in ${uids}; do count=`grep "$uid" test.log|wc -l` if [ "${count}" -ne "0" ]; then unique_uids[counter]="$uid" let counter=counter+1 fi done echo ${unique_uids[@]} echo $counter echo " Unique No:" ${#unique_uids[@]} echo uid StartTime EndTime" > $log for unique_uids in ${unique_uids[@]} ; do responseTime=`cat $i|grep "${unique_uids}" |awk '{split($2,Arr,":|,"); print Arr[1]*3600000+Arr[2]*60000+Arr[3]*1000+Arr[4]}'|sort -n` echo $unique_uids $responseTime >> $log done
And the result should be like this. The operation comes from id, the consumer comes from documentmetadata, and the hour is 08: 54: XX. Therefore, if we have more than one request and response, then we need to average the response time for requests at that hour.
Operation Consumer HOUR Avg-response-time (ms)
DKP_DumpDocumentProperties MRP 08 538
source share