☰

Apache CLFログファイルの一部のデータフィールドの概要

tag-icon • tag-icon shell-script text-processing scripting apache-httpd
Apache CLFログファイルの一部のデータフィールドの概要

生からいくつかの情報を収集したいです。結合ログ形式 Apache丸太：

51.254.56.62 - - [01/Jun/2016:20:49:28 +0500] "GET /vendors/jquery.slimscroll.min.js HTTP/1.1" 404 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:28 +0500] "GET /jquery.fullPage.js HTTP/1.1" 304 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/TweenLite.min.js HTTP/1.1" 304 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/EasePack.min.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/rAF.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
51.254.56.62 - - [01/Jun/2016:20:49:29 +0500] "GET /js/networkconfig.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:29 +0500] "GET /js/rAF.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:29 +0500] "GET /js/networkconfig.js HTTP/1.1" 304 - "http://networkconfig.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:28 +0500] "GET /vendors/jquery.slimscroll.min.js HTTP/1.1" 404 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
182.180.10.40 - - [01/Jun/2016:20:49:28 +0500] "GET /jquery.fullPage.js HTTP/1.1" 304 - "http://networkconfig.net/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"

これが私がしたことです：

  awk '{ print $1,$11}' accesslog | sort | uniq -c | sort -nr | head -n 10

  3 51.254.56.62 "http://networkconfig.net/"
  3 51.254.56.62 "http://networkconfig.com/"
  2 182.180.10.40 "http://networkconfig.net/"
  2 182.180.10.40 "http://networkconfig.com/"

私が得たいものは：

Domains                     Hits By IP

networkconfig.net           3 hits 51.254.56.62  | 2 hits 182.180.10.40 and so on
networkconfig.com           3 hits 51.254.56.62 | 2 hits 182.180.10.40 and so on

答え1

修正バージョン(3)中醜いシャン:

#!/bin/bash
{ echo "Domains  Hits by IP" ; \
  awk '{ print $1 gsub(/^.*:\/\/|\"|\/.*$/,"",$11) "\t" $11 }' $1 | \
      sort | \
      uniq  -c | \
      sort -k3,3 -k1,1nr | \
      while n="" read a b c; do \
          [ $a = 1 ] && p='' || p=s ; \
          if [ "$n" = "$c" ] ; then \
               echo -n "  |  $a hit$p $b" ; \
          else echo ; \
               echo -n   "$c $a hit$p $b" ; \
          fi  ; n="$c" ; \
      done ; \
      echo ; \
} | \
while read a b ; do \
    printf "%-30s   %s\n" "$a" "$b" ; \
done

出力./ugly.sh accesslog：

Domains                          Hits by IP

networkconfig.com                3 hits 51.254.56.62  |  2 hits 182.180.10.40
networkconfig.net                3 hits 51.254.56.62  |  2 hits 182.180.10.40

(OPのデータURL ./ugly.sh log.txt:ログ.txt):

Domains                          Hits by IP

-                                1 hit 180.76.15.138  |  1 hit 192.243.55.136
www.google.com.pk                3 hits 122.129.73.92
www.networkconfigorchard.com     2 hits 39.46.59.57  |  8 hits 39.46.6.0

関連情報