☰

巨大な順序のテキストファイルから2つの文字列間のテキストを抽出する

tag-icon • tag-icon text-processing grep sort
巨大な順序のテキストファイルから2つの文字列間のテキストを抽出する

我々は、数百万の順序付けられたタイムスタンプ観測を含む巨大なテキストファイルを持っており、始点と終点が与えられた場合は、次のものが必要です。クイックメソッドその期間の観測値を抽出します。

たとえば、以下はファイルの一部です。

"2018-04-05 12:53:00",28,13.6,7.961,1746,104.7878,102.2,9.78,29.1,0,2.432,76.12,955,38.25,249.9,362.4,281.1,0.04
"2018-04-05 12:54:00",29,13.59,7.915,1738,104.2898,102.2,10.01,29.53,0,1.45,200.3,952,40.63,249.3,361.4,281.1,0.043
"2018-04-05 12:55:00",30,13.59,7.907,1734,104.0326,102.2,10.33,28.79,0,2.457,164.1,948,41.39,249.8,361.3,281.1,0.044
"2018-04-05 12:56:00",31,13.59,7.937,1718,103.0523,102.2,10.72,31.42,0,1.545,8.22,941,42.06,249.4,361.1,281.1,0.045
"2018-04-05 12:57:00",32,13.59,7.975,1719,103.1556,102.2,10.68,29.26,0,2.541,0.018,940,41.95,249.1,360.1,281.1,0.045
"2018-04-05 12:58:00",33,13.59,8,1724,103.4344,102.2,10.35,29.58,0,1.908,329.8,942,42.65,249.5,361.4,281.1,0.045
"2018-04-05 12:59:00",34,13.59,8,1733,103.9831,102.2,10.23,30.17,0,2.59,333.1,948,42.21,250.2,362,281.2,0.045
"2018-04-05 13:00:00",35,13.59,7.98,1753,105.1546,102.2,10.17,29.06,0,3.306,332.4,960,42,250.4,362.7,281.1,0.044
"2018-04-05 13:01:00",36,13.59,7.964,1757,105.3951,102.2,10.24,30.75,0,2.452,0.012,962,42.03,250.4,362.4,281.1,0.044
"2018-04-05 13:02:00",37,13.59,7.953,1757,105.4047,102.2,10.31,31.66,0,3.907,2.997,961,41.1,250.6,362.4,281.1,0.043
"2018-04-05 13:03:00",38,13.59,7.923,1758,105.4588,102.2,10.28,29.64,0,4.336,50.19,962,40.85,250.3,362.6,281.1,0.042
"2018-04-05 13:04:00",39,13.59,7.893,1757,105.449,102.1,10.27,30.42,0,1.771,12.98,962,41.73,249.8,362.1,281.1,0.043
"2018-04-05 13:05:00",40,13.6,7.89,1757,105.4433,102.1,10.46,29.54,0,2.296,93.7,962,43.02,249.9,361.7,281,0.045
"2018-04-05 13:06:00",41,13.59,7.915,1756,105.3322,102.1,10.52,29.53,0,0.632,190.8,961,43.64,249.3,361.5,281,0.045
"2018-04-05 13:07:00",42,13.6,7.972,1758,105.4697,102.1,10.77,29.49,0,0.376,322.5,961,44.69,249.1,360.9,281.1,0.046
"2018-04-05 13:08:00",43,13.6,8.05,1754,105.233,102.1,11.26,28.66,0,0.493,216.8,959,44.8,248.4,360.1,281.2,0.047

「2018-04-05 13:00:00」と「2018-04-05 13:05:00」の間のデータポイントが必要な場合、出力は次のようになります。

"2018-04-05 13:00:00",35,13.59,7.98,1753,105.1546,102.2,10.17,29.06,0,3.306,332.4,960,42,250.4,362.7,281.1,0.044
"2018-04-05 13:01:00",36,13.59,7.964,1757,105.3951,102.2,10.24,30.75,0,2.452,0.012,962,42.03,250.4,362.4,281.1,0.044
"2018-04-05 13:02:00",37,13.59,7.953,1757,105.4047,102.2,10.31,31.66,0,3.907,2.997,961,41.1,250.6,362.4,281.1,0.043
"2018-04-05 13:03:00",38,13.59,7.923,1758,105.4588,102.2,10.28,29.64,0,4.336,50.19,962,40.85,250.3,362.6,281.1,0.042
"2018-04-05 13:04:00",39,13.59,7.893,1757,105.449,102.1,10.27,30.42,0,1.771,12.98,962,41.73,249.8,362.1,281.1,0.043
"2018-04-05 13:05:00",40,13.6,7.89,1757,105.4433,102.1,10.46,29.54,0,2.296,93.7,962,43.02,249.9,361.7,281,0.045

grepまたはsedなどの既存のツールは、awkソートされたファイルに適用するために最適化されていません。だから彼らは十分に高速ではありません。バイナリ検索を使用するツールは、この種の問題を解決するのに適しています。

答え1

非常に大きなファイルの場合、ユーティリティを使用すると、プレフィックスlookタイムスタンプの自然な順序を活用して、aと文字列の最大の共通プレフィックスに対して高速バイナリ検索を実行できます。その後、これを実行/後処理して、出力から関心のある行を抽出できます。startendawksedlook

存在するbash

export start='"2018-04-05 13:00:00"'
export end='"2018-04-05 13:05:00"'
#determine common prefix ("2018-04-05 13:0 in this example)
common_prefix=$(awk 'BEGIN {
   start=ENVIRON["start"]; end=ENVIRON["end"];
   len=length(start) > length(end)? length(end): length(start); 
   i=1;
   while (i <= len && substr(ENVIRON["start"], i, 1) == substr(ENVIRON["end"], i, 1)) {
       ++i
   }
    print(substr(start, 1, i-1))
}' </dev/null
)
#the -b option to look forces binary search. 
#My version of look on Ubuntu needs this flag to be passed, 
#some other versions of look perform a binary search by default and do not support a -b.
look -b "$common_prefix" file | awk '$0 ~ "^"ENVIRON["start"],$0 ~ "^"ENVIRON["end"]'

答え2

「2018-04-05 13:00:00」と「2018-04-05 13:05:00」の間の行を印刷します。

sed -n '/2018-04-05 13:00:00/,/2018-04-05 13:05:00/p' file

または

sed -n /"2018-04-05 13:00:00"/,/"2018-04-05 13:05:00"/p file

開始日「2018-04-05 13:00:00」をgrepし、次の5行（= 5分）を出力し、-m1最初の一致後に検索を停止します。

grep -m1 -A5 '2018-04-05 13:00:00' file

関連情報