.tif
私のディレクトリには約130000個のファイルがあります。 tesseractを使用してテキストを抽出する必要があります。メモリドライブを作成し、tmpfs
すべてのファイルをそこに移動しました。私の内部ページの1つは次のとおりです。
printf "%03d\n" {1..163} | parallel find . -name 'k-{}\*' -print0 | parallel -j100 -0 'tesseract {/} /mnt/ramdisk/output/{/.} > /dev/null 2>&1'
私のコンピュータは、4つのvCPUコアと16 GBのRAMを備えたUbuntu 16.04を実行するクラウド仮想マシンです。 100を超えるジョブを並列に実行していますが、1分あたり25から30を超えるファイルをインポートできません。
ディスクI / Oはボトルネックを引き起こしますか?私はこの主題についてほとんど何も知らない。毎分より多くのファイルを実行するにはどうすればよいですか?
ps -e
出力は次のとおりです。
21002 pts/0 00:00:06 tesseract <defunct>
21010 pts/0 00:00:05 tesseract <defunct>
21023 pts/0 00:00:06 tesseract <defunct>
21035 pts/0 00:00:04 tesseract <defunct>
21039 pts/0 00:00:06 tesseract <defunct>
21040 pts/0 00:00:04 tesseract <defunct>
21044 pts/0 00:00:00 tesseract <defunct>
21056 pts/0 00:00:04 tesseract <defunct>
21065 pts/0 00:00:05 tesseract <defunct>
21067 pts/0 00:00:06 tesseract <defunct>
21068 pts/0 00:00:06 tesseract <defunct>
21080 pts/0 00:00:06 tesseract <defunct>
21081 pts/0 00:00:06 tesseract <defunct>
21082 pts/0 00:00:05 tesseract <defunct>
21083 pts/0 00:00:06 tesseract <defunct>
21084 pts/0 00:00:06 tesseract <defunct>
21085 pts/0 00:00:00 tesseract <defunct>
21087 pts/0 00:00:06 tesseract <defunct>
21090 pts/0 00:00:05 tesseract <defunct>
21094 pts/0 00:00:00 tesseract <defunct>
21099 pts/0 00:00:05 tesseract <defunct>
21100 pts/0 00:00:02 tesseract <defunct>
21102 pts/0 00:00:06 tesseract <defunct>
21103 pts/0 00:00:05 tesseract <defunct>
21104 pts/0 00:00:06 tesseract <defunct>
21106 pts/0 00:00:08 tesseract <defunct>
21107 pts/0 00:00:06 tesseract <defunct>
21108 pts/0 00:00:04 tesseract <defunct>
21110 pts/0 00:00:06 tesseract <defunct>
21111 pts/0 00:00:08 tesseract <defunct>
21112 pts/0 00:00:06 tesseract <defunct>
21113 pts/0 00:00:04 tesseract <defunct>
21114 pts/0 00:00:06 tesseract <defunct>
21115 pts/0 00:00:06 tesseract <defunct>
21117 pts/0 00:00:01 tesseract <defunct>
21118 pts/0 00:00:00 tesseract <defunct>
21119 pts/0 00:00:04 tesseract <defunct>
21128 pts/0 00:00:05 tesseract <defunct>
21129 pts/0 00:00:00 tesseract <defunct>
21134 pts/0 00:00:06 tesseract <defunct>
21135 pts/0 00:00:04 tesseract <defunct>
21137 pts/0 00:00:06 tesseract <defunct>
21138 pts/0 00:00:00 tesseract <defunct>
21141 pts/0 00:00:06 tesseract <defunct>
21144 pts/0 00:00:05 tesseract <defunct>
21145 pts/0 00:00:06 tesseract <defunct>
21150 pts/0 00:00:06 tesseract <defunct>
21151 pts/0 00:00:02 tesseract <defunct>
21152 pts/0 00:00:06 tesseract <defunct>
21154 pts/0 00:00:00 tesseract <defunct>
21155 pts/0 00:00:05 tesseract <defunct>
21158 pts/0 00:00:05 tesseract <defunct>
21159 pts/0 00:00:06 tesseract <defunct>
21160 pts/0 00:00:06 tesseract <defunct>
21165 pts/0 00:00:06 tesseract <defunct>
21167 pts/0 00:00:05 tesseract <defunct>
21173 pts/0 00:00:00 tesseract <defunct>
21175 pts/0 00:00:00 tesseract <defunct>
21181 pts/0 00:00:06 tesseract <defunct>
21184 pts/0 00:00:05 tesseract <defunct>
21185 pts/0 00:00:06 tesseract <defunct>
21187 pts/0 00:00:05 tesseract <defunct>
21191 pts/0 00:00:06 tesseract <defunct>
21196 pts/0 00:00:00 tesseract <defunct>
21197 pts/0 00:00:05 tesseract <defunct>
21198 pts/0 00:00:06 tesseract <defunct>
21199 pts/0 00:00:08 tesseract
21200 pts/0 00:00:00 tesseract <defunct>
21201 pts/0 00:00:06 tesseract <defunct>
21202 pts/0 00:00:03 tesseract <defunct>
21204 pts/0 00:00:08 tesseract
21207 pts/0 00:00:06 tesseract <defunct>
21210 pts/0 00:00:00 tesseract <defunct>
21214 pts/0 00:00:00 tesseract <defunct>
21215 pts/0 00:00:06 tesseract <defunct>
21217 pts/0 00:00:05 tesseract <defunct>
21218 pts/0 00:00:03 tesseract <defunct>
21221 pts/0 00:00:06 tesseract <defunct>
21224 ? 00:00:00 kworker/0:1
21229 pts/0 00:00:06 tesseract <defunct>
21239 pts/0 00:00:06 tesseract <defunct>
21240 pts/0 00:00:04 tesseract <defunct>
21241 pts/0 00:00:06 tesseract <defunct>
21242 pts/0 00:00:02 tesseract <defunct>
21245 pts/0 00:00:06 tesseract <defunct>
21246 pts/0 00:00:05 tesseract <defunct>
21247 pts/0 00:00:04 tesseract <defunct>
21248 pts/0 00:00:00 tesseract <defunct>
21250 pts/0 00:00:06 tesseract <defunct>
21251 pts/0 00:00:05 tesseract <defunct>
21252 pts/0 00:00:01 tesseract <defunct>
21253 pts/0 00:00:06 tesseract <defunct>
21265 pts/0 00:00:06 tesseract <defunct>
21268 pts/0 00:00:06 tesseract <defunct>
21269 pts/0 00:00:06 tesseract <defunct>
21270 pts/0 00:00:01 tesseract <defunct>
21277 pts/0 00:00:06 tesseract <defunct>
21285 pts/0 00:00:05 tesseract
21289 pts/0 00:00:07 tesseract <defunct>
21291 pts/0 00:00:07 tesseract
21292 pts/0 00:00:06 tesseract
21293 pts/0 00:00:06 tesseract
21297 pts/0 00:00:06 tesseract
21299 pts/0 00:00:05 tesseract
21300 pts/0 00:00:06 tesseract <defunct>
21302 pts/0 00:00:05 tesseract
21303 pts/0 00:00:06 tesseract <defunct>
21308 pts/0 00:00:05 tesseract <defunct>
21320 pts/0 00:00:05 tesseract
21322 pts/0 00:00:00 tesseract <defunct>
21324 pts/0 00:00:05 tesseract <defunct>
21329 pts/0 00:00:05 tesseract <defunct>
21330 pts/0 00:00:05 tesseract <defunct>
21333 pts/0 00:00:05 tesseract
21334 pts/0 00:00:05 tesseract
21335 pts/0 00:00:06 tesseract
21337 pts/0 00:00:00 tesseract <defunct>
21338 pts/0 00:00:04 tesseract <defunct>
21340 pts/0 00:00:04 tesseract
21341 pts/0 00:00:05 tesseract
21344 pts/0 00:00:05 tesseract
21353 pts/0 00:00:06 tesseract <defunct>
21362 pts/0 00:00:06 tesseract <defunct>
21363 pts/0 00:00:00 tesseract <defunct>
21366 pts/0 00:00:03 tesseract
21372 pts/0 00:00:04 tesseract
21373 pts/0 00:00:04 tesseract
21377 pts/0 00:00:00 tesseract <defunct>
21378 pts/0 00:00:05 tesseract
21382 pts/0 00:00:05 tesseract <defunct>
21384 pts/0 00:00:04 tesseract
21385 pts/0 00:00:05 tesseract
21391 pts/0 00:00:05 tesseract <defunct>
21393 pts/0 00:00:01 tesseract <defunct>
21394 pts/0 00:00:04 tesseract
21396 pts/0 00:00:01 tesseract <defunct>
21397 pts/0 00:00:02 tesseract <defunct>
21399 pts/0 00:00:05 tesseract
21401 pts/0 00:00:06 tesseract <defunct>
21403 pts/0 00:00:00 tesseract <defunct>
21404 pts/0 00:00:01 tesseract <defunct>
21405 pts/0 00:00:00 tesseract <defunct>
21407 pts/0 00:00:04 tesseract
21408 pts/0 00:00:00 tesseract <defunct>
21409 pts/0 00:00:05 tesseract
21413 pts/0 00:00:02 tesseract <defunct>
21414 pts/0 00:00:04 tesseract
21415 pts/0 00:00:05 tesseract
21416 pts/0 00:00:04 tesseract
21417 pts/0 00:00:00 tesseract <defunct>
21418 pts/0 00:00:03 tesseract
21419 pts/0 00:00:03 tesseract
21422 pts/0 00:00:03 tesseract
21424 pts/0 00:00:04 tesseract
21426 pts/0 00:00:03 tesseract
21427 pts/0 00:00:04 tesseract
21429 pts/0 00:00:05 tesseract
21434 pts/0 00:00:01 tesseract <defunct>
21437 pts/0 00:00:04 tesseract
21438 pts/0 00:00:00 nest.sh
21440 pts/0 00:00:02 tesseract
21445 pts/0 00:00:03 tesseract
21446 pts/0 00:00:04 tesseract
21447 pts/0 00:00:04 tesseract
21448 pts/0 00:00:02 tesseract
21449 pts/0 00:00:03 tesseract
21450 pts/0 00:00:04 tesseract
21451 pts/0 00:00:04 tesseract
21452 pts/0 00:00:02 tesseract <defunct>
21454 pts/0 00:00:03 tesseract
21455 pts/0 00:00:03 tesseract
21457 pts/0 00:00:00 perl
21467 ? 00:00:00 kworker/2:2
21471 pts/0 00:00:03 tesseract
21472 pts/0 00:00:03 tesseract
21475 pts/0 00:00:00 tesseract <defunct>
21480 ? 00:00:00 kworker/u8:2
21485 pts/0 00:00:02 tesseract
21487 pts/0 00:00:03 tesseract
21488 pts/0 00:00:02 tesseract
21489 pts/0 00:00:03 tesseract
21493 pts/0 00:00:02 tesseract
21496 pts/0 00:00:03 tesseract
21498 pts/0 00:00:03 tesseract
21501 pts/0 00:00:03 tesseract
21508 pts/0 00:00:02 tesseract <defunct>
21510 pts/0 00:00:03 tesseract
21516 pts/0 00:00:00 tesseract <defunct>
21519 pts/0 00:00:02 tesseract
21521 pts/0 00:00:03 tesseract
21523 pts/0 00:00:02 tesseract
21531 pts/0 00:00:02 tesseract
21532 pts/0 00:00:02 tesseract
21536 pts/0 00:00:00 tesseract <defunct>
21537 pts/0 00:00:02 tesseract
21543 pts/0 00:00:01 tesseract
21545 pts/0 00:00:02 tesseract
21549 pts/0 00:00:02 tesseract
21553 pts/0 00:00:02 tesseract
21556 pts/0 00:00:00 tesseract <defunct>
21560 pts/0 00:00:00 tesseract
21561 pts/0 00:00:01 tesseract
21562 pts/0 00:00:01 tesseract
21566 pts/0 00:00:01 tesseract
21569 pts/0 00:00:01 tesseract
21570 pts/0 00:00:01 tesseract
21575 pts/0 00:00:00 tesseract <defunct>
21582 pts/0 00:00:02 tesseract
21583 pts/0 00:00:00 tesseract
21585 pts/0 00:00:01 tesseract
21586 pts/0 00:00:01 tesseract
21590 pts/0 00:00:01 tesseract
21595 pts/0 00:00:00 tesseract <defunct>
21596 pts/0 00:00:00 tesseract
21597 pts/0 00:00:01 tesseract
21598 pts/0 00:00:01 tesseract
21599 pts/0 00:00:00 tesseract
21600 pts/0 00:00:00 tesseract <defunct>
21606 pts/0 00:00:00 tesseract
21610 pts/0 00:00:00 tesseract
21611 pts/0 00:00:01 tesseract
21612 pts/0 00:00:00 tesseract
21615 pts/0 00:00:01 tesseract
21620 pts/0 00:00:01 tesseract
21621 pts/0 00:00:00 tesseract
21622 pts/0 00:00:00 tesseract
21624 pts/0 00:00:00 tesseract
21625 pts/0 00:00:00 tesseract
21626 pts/0 00:00:00 tesseract
21639 pts/0 00:00:01 tesseract
21641 pts/0 00:00:01 tesseract
21646 ? 00:00:00 mysql-systemd-s
21648 pts/0 00:00:00 tesseract
21653 pts/0 00:00:00 tesseract
21654 pts/0 00:00:00 tesseract
21655 pts/0 00:00:00 tesseract
21660 pts/0 00:00:00 tesseract
21665 pts/0 00:00:00 tesseract
21666 pts/0 00:00:00 tesseract
21670 pts/0 00:00:00 tesseract
21671 pts/0 00:00:00 tesseract
21674 pts/0 00:00:00 tesseract
21675 pts/0 00:00:00 tesseract
21677 pts/0 00:00:00 tesseract
21682 pts/0 00:00:00 tesseract
21683 pts/0 00:00:00 tesseract
21684 pts/0 00:00:00 tesseract
21688 pts/0 00:00:00 tesseract
21690 pts/0 00:00:00 tesseract
21691 pts/0 00:00:00 tesseract
21692 pts/0 00:00:00 tesseract
21693 pts/0 00:00:00 tesseract
21696 pts/0 00:00:00 tesseract
21697 pts/0 00:00:00 tesseract
21699 pts/0 00:00:00 tesseract
21703 pts/0 00:00:00 tesseract
21705 pts/0 00:00:00 tesseract
21708 pts/0 00:00:00 tesseract
21709 pts/0 00:00:00 tesseract
21710 pts/0 00:00:00 tesseract
21711 pts/0 00:00:00 tesseract
21712 pts/0 00:00:00 tesseract
21713 pts/0 00:00:00 tesseract
21714 pts/0 00:00:00 tesseract
21716 pts/0 00:00:00 tesseract
21717 pts/0 00:00:00 tesseract
21720 pts/0 00:00:00 tesseract
21724 pts/0 00:00:00 tesseract
21727 pts/0 00:00:00 tesseract
21730 pts/0 00:00:00 tesseract
21732 pts/0 00:00:00 tesseract
21733 pts/0 00:00:00 tesseract
21736 pts/0 00:00:00 tesseract
21739 pts/0 00:00:00 tesseract
21742 pts/0 00:00:00 tesseract
21745 pts/0 00:00:00 tesseract
21752 pts/0 00:00:00 tesseract
21755 pts/0 00:00:00 tesseract
21756 pts/0 00:00:00 tesseract
21758 pts/0 00:00:00 tesseract
21760 pts/0 00:00:00 tesseract
21762 pts/0 00:00:00 tesseract
21764 pts/0 00:00:00 tesseract
21765 pts/0 00:00:00 tesseract
21766 pts/0 00:00:00 tesseract
21769 pts/0 00:00:00 tesseract
21773 pts/0 00:00:00 tesseract
21778 pts/0 00:00:00 tesseract
21779 pts/0 00:00:00 tesseract
21780 pts/0 00:00:00 tesseract
21782 pts/0 00:00:00 tesseract
21784 pts/0 00:00:00 sleep
21785 pts/0 00:00:00 sleep
21786 pts/0 00:00:00 sleep
21787 pts/0 00:00:00 tesseract
21788 pts/0 00:00:00 bash
21789 pts/0 00:00:00 sleep
21790 pts/0 00:00:00 sleep
21791 pts/0 00:00:00 sleep
21792 pts/0 00:00:00 sleep
21794 pts/0 00:00:00 sleep
21795 pts/0 00:00:00 tesseract
21796 pts/0 00:00:00 tesseract
21801 pts/0 00:00:00 tesseract
21803 pts/0 00:00:00 bash
21804 pts/0 00:00:00 bash
21807 pts/0 00:00:00 sleep
21808 pts/0 00:00:00 tesseract
21809 pts/0 00:00:00 tesseract
21810 pts/0 00:00:00 bash
21811 pts/0 00:00:00 sleep
21812 pts/0 00:00:00 sleep
21813 pts/0 00:00:00 perl
21814 ? 00:00:00 sshd
21815 pts/0 00:00:00 perl
21816 ? 00:00:00 sleep
21817 pts/1 00:00:00 ps
答え1
すでにtmpfsを使用している場合は、IOによってボトルネックが発生しないでください。スクリプトの明らかな問題は、同時に多くのタスクを実行しようとしていることです。タスクはtmpfsのために何も待つべきではないので、保持しているコアの数(おそらく+ 1)と同じ数の並列タスクを使用する必要があります。また、より迅速にテストできるように、テストセットを200ファイルに減らします。
答え2
この入出力の原因を特定する必要があります。読み取りと書き込みはtmpfs
I / Oを発生しません。
:> cd /run/user/1000
:> LC_ALL=C df -h .
Filesystem Size Used Avail Use% Mounted on
tmpfs 2.4G 128M 2.3G 6% /run/user/1000
:> dd if=/dev/urandom of=bigfile bs=1M count=100
:> ls -lh bigfile
-rw-r--r-- 1 hl hauke 100M 30. Dez 16:07 bigfile
:> while true; do cat <bigfile >bigfilecopy || break; echo -n .; done
1GiB/s以上の速度でデータを読み書きしましたが、top
I/O待ち時間は0.0%となりました。
プロセスの開始は毎秒25回だけ発生し、とにかくページキャッシュによってオーバーライドされる必要があるため、問題はありません。
I / Oの原因を見つけて削除できない場合は、異なる少数の並列プロセスのI / O待機を比較する必要があります。すでに述べたように、一般にCPUコアよりも多くのプロセスを持つことは意味がありません。ただし、2 つまたは 3 つのプロセスが I/O レイテンシが非常に短く、4 つのプロセスが負荷が大幅に高くなる可能性があるため、3 つの並列プロセスを使用する方が高速な場合があります。