問題のサーバーはデータをレポートとして処理するために使用され、プロセスキューを確認するために毎分実行される3つのクローンジョブがあり、見つかったものがあればキューが空になるまで実行され続けます。特に忙しい日(過去3ヶ月間、火曜日などの奇妙な日)にはサーバーがダウンします。
これを実行すると、ps aux | grep 'httpd' | awk '{print $6/1024 " MB";}'
3つのhttpdワーカーがリニアな方法でメモリを取得し続けることを確認できます。レポートの数が多いため、プロセスは終了せず、引き続きメモリを増やします。
これはPHPランタイムであり、内部メモリ使用量を調べてみるとまったく問題にならないことがわかります。 PHP メモリのハード制限は 256 MB であり、PHP 階層内では、レポート処理が完了するとメモリが最小解放されます。現在、冗長サーバーでPHPメモリアナライザを使用して、この問題を引き起こしているPHP層に問題があるかどうかを確認してください。ただし、上記のps auxコマンドは、httpdワーカースレッドのメモリが約270 MBから始まり、サーバー全体(8 GB RAM)が使い果たされるまで増加し続けることを示しています。そのため、httpdはPHPプロセスやシステムにリサイクルするのではなく、PHP層が使用するすべてのメモリを占有できるようです。以下は、この問題が発生したときに発生するエラーログの一部です。
[Tue Dec 04 09:20:34.805175 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:20:45.142735 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
mmap() failed: [
mmap() failed: [
mmap() failed: [1122]] CCannot allocate memoryannot allocate memory12] Cannot allocate memory
mmap() failed: [
mmap() failed: [1122]] CCannot allocate memoryannot allocate memory
mmap() failed: [12] Cannot allocate memory[Tue Dec 04 09:20:45.764357 2018] [php7:error] [pid 12198] [client 127.0.0.1:37694] PHP Fatal error:
Out of memory (allocated 2097152) (tried to allocate 65536 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440
[Tue Dec 04 09:20:45.764411 2018] [php7:error] [pid 12197] [client 127.0.0.1:37692] PHP Fatal error: Out of memory (allocated 2097152) (tried
to allocate 65536 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440
[Tue Dec 04 09:20:45.764358 2018] [php7:error] [pid 12191] [client 127.0.0.1:37690] PHP Fatal error: Out of memory (allocated 2097152) (tried
to allocate 65536 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440
mmap() failed: [
mmap() failed: [1122]] CCannot allocate memoryannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [
mmap() failed: [1122]] CCannot allocate memoryannot allocate memory
mmap() failed: [12] Cannot allocate memory[Tue Dec 04 09:20:46.257438 2018] [php7:error] [pid 12191] [client 127.0.0.1:37690] PHP Fatal error:
Out of memory (allocated 2097152) (tried to allocate 73728 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440
[Tue Dec 04 09:20:46.257439 2018] [php7:error] [pid 12197] [client 127.0.0.1:37692] PHP Fatal error: Out of memory (allocated 2097152) (tried
to allocate 73728 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440
[Tue Dec 04 09:20:46.257481 2018] [php7:error] [pid 12198] [client 127.0.0.1:37694] PHP Fatal error: Out of memory (allocated 2097152) (tried
to allocate 73728 bytes) in /var/www/html/vendor/composer/ClassLoader.php on line 440
[Tue Dec 04 09:20:57.249945 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:23:08.171314 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:24:33.415351 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:26:22.308600 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:27:55.057324 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:29:17.174173 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:30:39.193341 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:33:16.023329 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:34:53.208958 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:35:30.902310 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:36:05.215192 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:37:36.511811 2018] [mpm_prefork:error] [pid 2751] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Dec 04 09:42:18.453045 2018] [suexec:notice] [pid 2699] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
使用中のhttpdモード:
$ sudo httpd -M
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
allowmethods_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_anon_module (shared)
authn_core_module (shared)
authn_dbd_module (shared)
authn_dbm_module (shared)
authn_file_module (shared)
authn_socache_module (shared)
authz_core_module (shared)
authz_dbd_module (shared)
authz_dbm_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_owner_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cache_module (shared)
cache_disk_module (shared)
cache_socache_module (shared)
data_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
dumpio_module (shared)
echo_module (shared)
env_module (shared)
expires_module (shared)
ext_filter_module (shared)
filter_module (shared)
headers_module (shared)
http2_module (shared)
include_module (shared)
info_module (shared)
log_config_module (shared)
logio_module (shared)
macro_module (shared)
mime_magic_module (shared)
mime_module (shared)
negotiation_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
request_module (shared)
rewrite_module (shared)
setenvif_module (shared)
slotmem_plain_module (shared)
slotmem_shm_module (shared)
socache_dbm_module (shared)
socache_memcache_module (shared)
socache_shmcb_module (shared)
status_module (shared)
substitute_module (shared)
suexec_module (shared)
unixd_module (shared)
userdir_module (shared)
version_module (shared)
vhost_alias_module (shared)
watchdog_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
lua_module (shared)
mpm_prefork_module (shared)
proxy_module (shared)
lbmethod_bybusyness_module (shared)
lbmethod_byrequests_module (shared)
lbmethod_bytraffic_module (shared)
lbmethod_heartbeat_module (shared)
proxy_ajp_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_fdpass_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_hcheck_module (shared)
proxy_scgi_module (shared)
proxy_uwsgi_module (shared)
proxy_wstunnel_module (shared)
ssl_module (shared)
php7_module (shared)
$ httpd -l
Compiled in modules:
core.c
mod_so.c
http_core.c
httpd/http.d/http.conf
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 4
MaxRequestWorkers 10
MaxConnectionsPerChild 1
</IfModule>
答え1
...プロセスキューを確認するために、1分ごとに3つのcronジョブが実行されます。見つかったエントリがある場合は、キューが空になるまで実行され続けます。
これによりメモリリークが発生しやすくなるため、bash 悪魔を使用してキュー内の各エントリの新しいインスタンスを順番に作成します。
while true ; do
while read -r F ; do
php /var/www/html/my.site.com/process1.php "$F"
done < <(find /path/to/queue -type f)
sleep 1
done
呼ぶ
wget -q -O /dev/null http://localhost/core/processing
不必要にApacheを含め、エントリ期間よりも長い間メモリを保持することを許可します。