☰

awkを使用してタイムスタンプと値をグループ化する方法

tag-icon • tag-icon shell-script awk scripting
awkを使用してタイムスタンプと値をグループ化する方法

分と値でグループ化する必要がある複雑なログがあります。以下は、いくつかのサンプルログです。

2019-08-09T19:01:53:594+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test1","logTime":"2019-08-09T12:01:53.594Z","responseTime":4}
2019-08-09T19:01:53:673+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test2","logTime":"2019-08-09T12:01:53.673Z","responseTime":4}
2019-08-09T19:14:03:773+07:00 - info: {"tag":"request /validate","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"error internal"},"metadata":"test3","logTime":"2019-08-09T12:14:03.773Z","responseTime":7}
2019-08-09T19:19:32:925+07:00 - info: {"tag":"request /validate","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"error internal"},"metadata":"test4","logTime":"2019-08-09T12:19:32.925Z","responseTime":8}

私の期待は次のとおりです。

19:01  errMessage : connect ECONNREFUSED 127.0.0.1:7000  10
19:02  errMessage : error internal  20
19:03  errMessage : error internal  10

有名な：

19:01 = hour minutes
errMessage : error internal = value
20 = count message err

以下のawkを試しましたが、まだ結果をグループ化していません。

cat file.log | strings | grep "errMessage" | awk -F'[{,]' '{print $1,$3,$4,$5,$8}' | awk -F'[-,"]' '{print $3,$11,$12,$13,$15,$16,$17}'

タイムスタンプとカウント値に基づいて結果をグループ化する方法を見つけるのに役立ちますか？

ありがとう

答え1

質問に提供されたデータがあまりにも希薄で拡張されました。これにより、グループ化ロジックを分単位、エラーメッセージ、および数でよりよく実証/検証できます。

2019-08-09T19:02:00:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test1","logTime":"2019-08-09T12:02:00.000Z","responseTime":4}
2019-08-09T19:02:03:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test1","logTime":"2019-08-09T12:02:00.000Z","responseTime":4}
2019-08-09T19:02:10:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test2","logTime":"2019-08-09T12:02:10.000Z","responseTime":4}
2019-08-09T19:02:15:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Internal Server Error","data":{},"errMessage":"TypeError: Cannot read property 'name' of undefined"},"metadata":"test2","logTime":"2019-08-09T12:02:10.000Z","responseTime":10}
2019-08-09T19:02:20:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test3","logTime":"2019-08-09T12:02:20.000Z","responseTime":4}
2019-08-09T19:02:25:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test3","logTime":"2019-08-09T12:02:20.000Z","responseTime":4}
2019-08-09T19:02:30:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test4","logTime":"2019-08-09T12:02:30.000Z","responseTime":4}
2019-08-09T19:02:35:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Internal Server Error","data":{},"errMessage":"ReferenceError: foo is not defined"},"metadata":"test4","logTime":"2019-08-09T12:02:30.000Z","responseTime":20}
2019-08-09T19:02:40:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test5","logTime":"2019-08-09T12:02:40.000Z","responseTime":4}
2019-08-09T19:02:45:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test5","logTime":"2019-08-09T12:02:40.000Z","responseTime":4}
2019-08-09T19:02:50:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test6","logTime":"2019-08-09T12:02:50.000Z","responseTime":4}
2019-08-09T19:02:55:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test6","logTime":"2019-08-09T12:02:50.000Z","responseTime":4}
2019-08-09T19:03:00:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test7","logTime":"2019-08-09T12:03:00.000Z","responseTime":4}
2019-08-09T19:03:05:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test7","logTime":"2019-08-09T12:03:00.000Z","responseTime":4}
2019-08-09T19:03:10:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test8","logTime":"2019-08-09T12:03:10.000Z","responseTime":4}
2019-08-09T19:03:15:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Internal Server Error","data":{},"errMessage":"SyntaxError: Unexpected token ':'"},"metadata":"test8","logTime":"2019-08-09T12:03:10.000Z","responseTime":15}
2019-08-09T19:03:20:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test9","logTime":"2019-08-09T12:03:20.000Z","responseTime":4}
2019-08-09T19:03:25:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test9","logTime":"2019-08-09T12:03:20.000Z","responseTime":4}
2019-08-09T19:03:30:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test10","logTime":"2019-08-09T12:03:30.000Z","responseTime":4}
2019-08-09T19:03:35:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Internal Server Error","data":{},"errMessage":"TypeError: Cannot read property 'length' of undefined"},"metadata":"test10","logTime":"2019-08-09T12:03:30.000Z","responseTime":25}
2019-08-09T19:03:45:000+07:00 - info: {"tag":"request /browse","tagName":{"status":"05","message":"Unrecognise Request","data":{},"errMessage":"connect ECONNREFUSED 127.0.0.1:7000"},"metadata":"test11","logTime":"2019-08-09T12:03:40.000Z","responseTime":4}

これまで私TXR不明瞭な音声プログラムは次のような結果を生成します。

$ txr group.tl  < xlog
19:02  errMessage : connect ECONNREFUSED 127.0.0.1:7000  10
19:02  errMessage : ReferenceError: foo is not defined  1
19:02  errMessage : TypeError: Cannot read property 'name' of undefined  1
19:03  errMessage : connect ECONNREFUSED 127.0.0.1:7000  7
19:03  errMessage : SyntaxError: Unexpected token ':'  1
19:03  errMessage : TypeError: Cannot read property 'length' of undefined  1

プログラムは次のとおりgroup.tlです。

(defstruct log ()
  hour minute
  err-message)

(build
  (let (minute-batch cur-hh cur-mm)
    (flet ((flush ()
             (let ((by-message-hash (group-by .err-message (get))))
               (oust)
               (dohash (msg group by-message-hash)
                 (let ((lead (first group)))
                   (put-line `@{lead.hour}:@{lead.minute}  errMessage : @msg  @(len group)`))))))
      (whilet ((line (get-line)))
        (when-match `@{nil}T@hh:@mm:@nil - info: @json` line
          (let ((jobj (get-json json)))
            (if (or (nequal hh cur-hh)
                    (nequal mm cur-mm))
              (flush))
            (set cur-hh hh cur-mm mm)
            (add (new log
                      hour hh minute mm
                      err-message [[jobj "tagName"] "errMessage"])))))
      (flush))))

このプログラムは2つのトップレベルの形式で構成されています。

  • aは、時間とエラーメッセージに関する情報を保持する構造をdefstruct定義します。log
  • buildすべてのロジックを含む式です。

このbuildマクロには、プログラム構成とリスト操作のための環境が含まれています。多くの地域オペレーターを提供しています。ここでは3つを使います。add暗黙的なリストに項目を追加します。getリストを検索します。oustリストを別のリストに置き換えます（引数が与えられていない場合、デフォルトは空です）。

戦略は、データをスキャンして各行をオブジェクトに変換することですlog。時間または分が変更されるたびに、累積グループをflush処理するためにローカル関数を呼び出します。累積オブジェクトのリストを検索し、リストを消去するflushために使用されます。入力が足りない場合は、もう一度呼び出して最後のバッチを取得します。(get)log(oust)flush

このflush関数は、エラーメッセージごとにオブジェクトをグループ化してハッシュテーブルを形成し、情報をダンプします。各エラーメッセージについて、時間、エラーメッセージ、およびエラーメッセージを共有したグループのサイズを印刷します。

項目の解析は、パターンマッチングを使用して実行されます。さらに、JSON部分を1つの単位として抽出し、TXR LispでJSONパーサーを使用します。 JSONオブジェクトはハッシュテーブルとして作成されます。 2つのレベルのテーブルを介して到着しますerrMessage。

関連情報