☰

ファイルの特定の行だけをフィルタリングする方法は？

tag-icon • tag-icon awk grep cut json
ファイルの特定の行だけをフィルタリングする方法は？

次の行を含むファイルがあります。最初の行に国のIPアドレスを、2行目に1回だけ、スコアラインを1回だけ見たいのですが、スコアラインが最も高い値（この場合は7.1）でなければなりません。

{
  "ip": "86.75.227.72",
  "history": [
    {
     "created": "2012-03-22T07:26:00.000Z",
     "reason": "Regional Internet Registry",
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.64.0.0/12",
     "categoryDescriptions": {},
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1,
     "cats": {}
   },
   {
     "created": "2012-04-13T13:34:00.000Z",
     "reason": "DNS heuristics",
     "cats": {
       "Dynamic IPs": 100
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.64.0.0/12",
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1
   },
   {
     "created": "2014-01-22T19:08:00.000Z",
     "reason": "DNS heuristics",
     "cats": {
       "Dynamic IPs": 86
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.72.0.0/14",
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1
   },
   {
     "created": "2014-03-09T13:11:00.000Z",
     "reason": "DNS heuristics",
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1
   },
   {
     "created": "2017-07-26T06:24:00.000Z",
     "reason": "Regional Internet Registry",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1
   },
   {
     "created": "2017-10-10T06:23:00.000Z",
     "reason": "Regional Internet Registry",
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1
   },
   {
     "created": "2017-10-18T06:23:00.000Z",
     "reason": "Regional Internet Registry",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.224.0/21",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1
   },
   {
     "created": "2017-11-20T18:16:00.000Z",
     "reason": "Third party feed",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.227.72/32",
     "cats": {
       "Dynamic IPs": 71,
       "Bots": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines.",
       "Bots": "IPs known for botnet-member activity. Devices using these IPs are obviously infected and take part in DDoS-attacks, port-scanning, spam-sending etc."
     },
     "reasonDescription": "This data was imported from a third party feed.",
     "score": 7.1
   },
   {
     "created": "2017-11-25T21:46:00.000Z",
     "reason": "Third party feed",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.75.227.72/32",
     "cats": {
       "Dynamic IPs": 71
     },
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "This data was imported from a third party feed.",
     "score": 1
   }
 ],
 "subnets": [
   {
     "created": "2017-10-18T06:23:00.000Z",
     "reason": "Regional Internet Registry",
     "asns": {
       "15557": {
         "Company": "LDCOMNET, FR",
         "cidr": 12
       }
     },
     "geo": {
       "country": "France",
       "countrycode": "FR"
     },
     "ip": "86.64.0.0",
     "categoryDescriptions": {},
     "reasonDescription": "One of the five RIRs announced a (new) location mapping of the IP.",
     "score": 1,
     "cats": {},
     "subnet": "86.64.0.0/12"
   },
   {
     "created": "2014-03-09T13:11:00.000Z",
     "reason": "DNS heuristics",
     "cats": {
       "Dynamic IPs": 71
     },
     "ip": "86.75.224.0",
     "categoryDescriptions": {
       "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
     },
     "reasonDescription": "Based on statistical DNS analysis.",
     "score": 1,
     "subnet": "86.75.224.0/21"
   }
 ],
 "cats": {
   "Dynamic IPs": 71
 },
 "geo": {
   "country": "France",
   "countrycode": "FR"
 },
 "score": 1,
 "reason": "Third party feed",
 "reasonDescription": "This data was imported from a third party feed.",
 "categoryDescriptions": {
   "Dynamic IPs": "This category contains IP addresses of dialup hosts and DSL lines."
 },
 "tags": []
}

「ボット」：「ボットネットメンバーがアクティブであることが知られているIPです。これらのIPを使用するデバイスは感染してDDoS攻撃に参加しているようです。

"score":7.1}
"geo":{"country":"France"
"score":1}]
"geo":{"country":"France"
"score":1
"score":1
"geo":{"country":"France"
"score":1

答え1

$ jq -r '.history | max_by(.score) | .ip' file.json
86.75.227.72/32

これは、配列内の最大値を持つ項目を見つけるために使用されますjq。見つかったら、見つかったアイテムから値が抽出されます。.history.score.ip

出力形式は若干異なります。ここでは、CSVを使用してIPアドレス、国、会社名（利用可能な場合）、およびスコアを配列として出力します.history。

$ jq -r '.history[] | [.ip, .geo.country, .asns."15557".Company, .score] | @csv' file.json
"86.64.0.0/12","France",,1
"86.64.0.0/12","France",,1
"86.72.0.0/14","France",,1
"86.75.224.0/21","France",,1
"86.75.224.0/21","France","LDCOMNET, FR",1
"86.75.224.0/21","France",,1
"86.75.224.0/21","France","LDCOMNET, FR",1
"86.75.227.72/32","France","LDCOMNET, FR",7.1
"86.75.227.72/32","France","LDCOMNET, FR",1

これと同じですが、最も高いスコアを持つアイテムのみが選択されます。

$ jq -r '.history | max_by(.score) | [.ip, .geo.country, .asns."15557".Company, .score] | @csv' file.json
"86.75.227.72/32","France","LDCOMNET, FR",7.1

答え2

離れてawkいる:

$ awk '$1=="\"ip\":"{
        ip=$2
       }
       $1=="\"country\":"{
        c[ip]=$2
       }
       $1=="\"score\":" && s[ip]<$2{ 
         s[ip]=$2
       }
       END{
           for(ip in c){
            print ip,c[ip],s[ip]
           }
       }' file 
"86.72.0.0/14", "France", 1
"86.64.0.0/12", "France", 1,
"86.75.224.0/21", "France", 1
"86.75.227.72/32", "France", 7.1
"86.75.227.72", "France", 
"86.75.224.0", "France", 1,

または、最高のスコアのIPだけを望み、他のものは望ましくない場合：

$ awk '$1=="\"ip\":"{ip=$2}$1=="\"score\":" && score<$2{score=$2;sip=ip}END{print sip} ' file 
"86.75.227.72/32"

関連情報