☰

awkを使用したsyslogデータの解析と書式設定

tag-icon • tag-icon awk
awkを使用したsyslogデータの解析と書式設定

ジェネレータからsyslogデータを取得していますが、行の先頭の日付と時刻とファイルの終わりのタスクを抽出したいと思います。

これまで私が持っていたのはこれだけです。

awk -F ' ' '{print $1,$2,$3,":",$NF}' test.log

私の出力に不完全なログメッセージがあります。

Sep 17 20:47:02 : Warning
Sep 20 12:02:20 : Started
Sep 20 12:09:31 : Level
Sep 20 12:10:27 : Level
Sep 20 12:10:28 : Stopped
Sep 23 09:19:03 : Auto
Sep 23 09:19:04 : Cleared
Sep 23 09:19:13 : Auto
Sep 30 12:33:48 : Auto
Sep 30 12:33:55 : Started
Sep 30 12:41:14 : Level
Sep 30 12:41:56 : Stopped
Sep 30 12:41:58 : Cleared
Oct 3 14:00:52 : Started
Oct 3 14:15:17 : Stopped
Oct 3 14:20:39 : Auto
Oct 15 12:02:55 : Started
Oct 15 12:12:50 : Stopped

これは実際の日付と時刻と完全なジョブメッセージを印刷したいものです。

Date Time Fuel has High Warning
Date Time Genset Started
Date Time  Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Date Time  Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Date Time  Genset Stopped
Date Time  Genset Not In Auto
Date Time  All Faults Cleared
Date Time  Genset in Auto
Date Time  Genset Not In Auto
Date Time  Genset Started
Date Time  Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Date Time  Genset Stopped
Date Time  All Faults Cleared
Date Time  Genset Started
Date Time  Genset Stopped
Date Time  Genset in Auto
Date Time  Genset Started
Date Time  Genset Stopped

各行には固定長がありません。

テストファイルです

Sep 17 20:47:02 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#011#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/17/2023 8:47:01 PM0#023#006#013+#006#001#004#001��s#001#002#000#004#004Fuel0#023#006#013+#006#001#004#001��s#001#004#000#004#00445940#026#006#013+#006#001#004#001��s#001#003#000#004#007Warning0$#006#013+#006#001#004#001��s#001#005#000#004#025Fuel has High Warning
Sep 20 12:02:20 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#012#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:02:23 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Sep 20 12:09:31 192.168.22.60 0�#001(#002#001#001#004#005AE4ML��#001#032#002#001#013#002#001#000#002#001#0000�#001#0150#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:09:34 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#022#006#013+#006#001#004#001��s#001#004#000#004#0031460#026#006#013+#006#001#004#001��s#001#003#000#004#007Warning0s#006#013+#006#001#004#001��s#001#005#000#004dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 192.168.22.60 0�#001##002#001#001#004#005AE4ML��#001#025#002#001#014#002#001#000#002#001#0000�#001#0100#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:10:30 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#022#006#013+#006#001#004#001��s#001#004#000#004#0031510#027#006#013+#006#001#004#001��s#001#003#000#004#010Shutdown0m#006#013+#006#001#004#001��s#001#005#000#004^Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Sep 20 12:10:28 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#015#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:10:30 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped
Sep 23 09:19:03 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#016#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/23/2023 9:19:05 AM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451380#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022Genset Not In Auto
Sep 23 09:19:04 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#017#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/23/2023 9:19:07 AM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00472190#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022All Faults Cleared
Sep 23 09:19:13 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#020#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/23/2023 9:19:16 AM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00456640#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset in Auto
Sep 30 12:33:48 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#021#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:33:50 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451380#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022Genset Not In Auto
Sep 30 12:33:55 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#022#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:33:57 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Sep 30 12:41:14 192.168.22.60 0�#001(#002#001#001#004#005AE4ML��#001#032#002#001#023#002#001#000#002#001#0000�#001#0150#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:41:17 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#022#006#013+#006#001#004#001��s#001#004#000#004#0031460#026#006#013+#006#001#004#001��s#001#003#000#004#007Warning0s#006#013+#006#001#004#001��s#001#005#000#004dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 30 12:41:56 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#024#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:41:59 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped
Sep 30 12:41:58 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#025#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:42:01 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00472190#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022All Faults Cleared
Oct  3 14:00:52 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#026#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#02410/3/2023 2:00:55 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Oct  3 14:15:17 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#027#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#02410/3/2023 2:15:20 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped
Oct  3 14:20:39 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#030#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#02410/3/2023 2:20:42 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00456640#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset in Auto
Oct 15 12:02:55 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#031#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000%#006#013+#006#001#004#001��s#001#001#000#004#02610/15/2023 12:02:59 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Oct 15 12:12:50 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#032#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000%#006#013+#006#001#004#001��s#001#001#000#004#02610/15/2023 12:12:54 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped

答え1

データに無効な文字セットエンコーディングを使用しているようです。 （これはデータの文字を説明できます。）処理中の出力に関する参照文書を調べて、出力がどのように解釈されるべきかを確認する必要があります。

ただし、「基本的に正確」であることを考慮すると、フィールド区切り文字は基本的に次のようになる#ため、次のように開始できます。

awk -F'#' '{ print $1, $NF }'

サンプル出力

Sep 17 20:47:02 192.168.22.60 0�� 025Fuel has High Warning
Sep 20 12:02:20 192.168.22.60 0�� 016Genset Started
Sep 20 12:09:31 192.168.22.60 0� 004dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 192.168.22.60 0� 004^Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level

これで、スペースで区切られた最初のフィールドセットを分割し、経験的に「最後」フィールドの先頭から3文字を削除する必要があります。

awk -F'#' '{ split($1, a, " "); print a[1], a[2], a[3], substr($NF,4,length($NF)) }'

サンプル出力

Sep 17 20:47:02 Fuel has High Warning
Sep 20 12:02:20 Genset Started
Sep 20 12:09:31 dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 ^Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level

dこれを維持するために、3行目と4行目から先行するものと削除する方法と理由の明確な証拠はありません。^

答え2

POSIX awkを使用してください。

$ awk '{sub(/[0-9]+\..*(#[0-9]{3}){4}[^[:upper:]]*/,"")} 1' file
Sep 17 20:47:02 Fuel has High Warning
Sep 20 12:02:20 Genset Started
Sep 20 12:09:31 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Sep 20 12:10:28 Genset Stopped
Sep 23 09:19:03 Genset Not In Auto
Sep 23 09:19:04 All Faults Cleared
Sep 23 09:19:13 Genset in Auto
Sep 30 12:33:48 Genset Not In Auto
Sep 30 12:33:55 Genset Started
Sep 30 12:41:14 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 30 12:41:56 Genset Stopped
Sep 30 12:41:58 All Faults Cleared
Oct  3 14:00:52 Genset Started
Oct  3 14:15:17 Genset Stopped
Oct  3 14:20:39 Genset in Auto
Oct 15 12:02:55 Genset Started
Oct 15 12:12:50 Genset Stopped

#NNN上記は、削除したいセクションの最後に常に少なくとも4つの文字列シーケンスがあり、#NNN印刷したい残りのセクションには4つの文字列はありませんが、#他のコンテキストではsを含めることができると仮定しています。 。

dさらに、印刷されるジョブの一部は常に大文字で始まると仮定する。そうしないと、予想される出力に基づいて入力の3行目の先頭から削除する必要があることがわかりません。dEngine ...

-Eたとえば、EREをサポートするsed（GNUまたはBSD sedなど）を使用してsedで同じことを実行できます。

$ sed -E 's/[0-9]+\..*(#[0-9]{3}){4}[^[:upper:]]*//' file
Sep 17 20:47:02 Fuel has High Warning
Sep 20 12:02:20 Genset Started
Sep 20 12:09:31 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Sep 20 12:10:28 Genset Stopped
Sep 23 09:19:03 Genset Not In Auto
Sep 23 09:19:04 All Faults Cleared
Sep 23 09:19:13 Genset in Auto
Sep 30 12:33:48 Genset Not In Auto
Sep 30 12:33:55 Genset Started
Sep 30 12:41:14 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 30 12:41:56 Genset Stopped
Sep 30 12:41:58 All Faults Cleared
Oct  3 14:00:52 Genset Started
Oct  3 14:15:17 Genset Stopped
Oct  3 14:20:39 Genset in Auto
Oct 15 12:02:55 Genset Started
Oct 15 12:12:50 Genset Stopped

関連情報