最近服务器的 httpd <defunct> 又频繁增多,主要原因现在还在寻找中。今天通宵看了很多文摘,同时也有所心得。现在把整个过程记录一下，以备后用。
第一步，针对log 文件的2G 的limitation 作排除性工作。(关于 2G log 的讨论请看[internetworkers] httpd 。但是这个 bug 已经被修补。)
这一步也就是清除以前的log文件，使用新的 cronolog 来使log文件进行循环记录。前几天已经对 cronolog 作了介绍，这里仅仅把在 httpd.conf 和 awstats.hostname.conf中修改的部分指摘一下。
建立新 log 目录
# mkdir /usr/local/apache2/log/weblog.kreny.com/
httpd.conf 中的 log 部分的修改
ErrorLog "|/usr/local/sbin/cronolog /usr/local/apache2/logs/weblog.kreny.com/weblog.kreny.com-error_log.%Y-%m"
CustomLog "|/usr/local/sbin/cronolog /usr/local/apache2/logs/weblog.kreny.com/weblog.kreny.com_log.%Y-%m" combined env=!LIMITIP
awstats.weblog.conf中的修改
LogFile="/usr/local/apache2/logs/weblog.kreny.com/weblog.kreny.com-access_log.%YYYY-1-%MM-1"
备份好awstats的数据目录，在我的主机里面是 /var/lib/awstats/
这样就排除了 2G log 的限制而导致 httpd 的问题。
第二步，由 PHP script 带来的一些修改。
查看了 apache 的 error log文件，发现有以下错误信息。
[client 61.135.145.216] PHP Warning: mysql_errno(): supplied argument is not a valid MySQL-Link resource in /home/hzmjp/public_html/phpbb2.1.1_patched_from_2.1.0/db/mysql4.php on line 331
[client 202.121.192.91] PHP Warning: mysql_connect(): #HY000Host 'dalouis.com' is not allowed to connect to this MySQL server in /home/hzmjp/public_html/phpbb2.1.1_patched_from_2.1.0/db/mysql4.php on line 48, referer: http://bbs.hzmjp.com
PHP Warning: mysql_connect(): Too many connections ... <这个是我在用 httperf测试时出现的，暂且也算一个问题吧...>
这里首先对 mysql_connect(): Too many connections 进行修正。
我google了一下后，以下这篇文章给我很多hints：
PHAkt Forum :: mysql_connect() too many connections??? URGENT!!
其主要原因是 mysql_pconnect() 的 mysql_connect() 的互用问题。于是查询了错误所在行，即 phpbb 2.1 的 ./db/mysql4.php 中的第 48 行。将
$this->db_connect_id = ($this->persistency) ? mysql_pconnect($this->server, $this->user, $this->password) : mysql_connect($this->server, $this->user, $this->password);
改为了：
$this->db_connect_id = ($this->persistency) ? mysql_connect($this->server, $this->user, $this->password) : mysql_connect($this->server, $this->user, $this->password);
同时，根据 MySQL Manual 中关于 Too many connections 此的描述，修改了 my.cnf 等一些设置。
第三步，对 MySQL 的 my.cnf 和 php.ini 的一些修改。
首先添加了改一下 my.cnf 中的设置
max_connections = 300
同时我也查询了一下 MySQL 关于 performance 的问题。使用一下语句可以查询 MySQL 的现状：
[root@dalouis awstats]# mysqladmin extended -u root -p
Enter password:
请看接下来这步是凭感觉做的，即修改 php.ini 中关于 memory 的部分：
memory_limit = 8M (原先我设置了30M)
第四步，优化一下 httpd.conf 中关于 works.c 的配置。
这点曾经已经讨论地很透彻了。又一遍通读了 apache.org MPM works 的 How it Works 和 fei 的 Apache 2.0性能优化—MPM的选择与配置。 现在把整个 httpd.conf 的 works.c 部分帖一下,里面有详细的解释，也请大家指教了。
我的配置 works.c.txt 和默认的配置 works.c.def.txt
第五步，关于 FastCGI 和 Keepalive 的一些考虑。
因为在实际操作中，往往在打开诸如 MT 和 awstats 这类 cgi 和 pl 网页时（网页比较长，同时对 cpu 的要求大），会出现打到一半打不开(此时 httpd <defunct> 出现可能性很大)的情况。所以关于 FastCGI 之类的资料也搜索了一下。
看了 CU 上的 Apache的进程与线程讨论(请忽视此帖上关于类似于 ServerLimit 等的讨论，这样只会把自己的思路和理解搞混)，发现了 mmHunter 的回复：
首先，如果你想用php的话，就不要用worker方式的mpm了，因为php到现在还没有保证所有的代码都是线程安全的如果你真的这么有兴趣试验Apache2+worker+php，可以考虑fastcgi方式的php, 具体可以参考fastcgi.coremail.cn
现在对这个还没有来得及研究，不过可以给人很多hint.
第六步，检测 apache 的状态。
使用 Server-Status 来检测 apache 的状态。注意，需要在 httpd.conf 中将 ExtendedStatus On 才能实施检测。
在 httpd.conf 中这样设置：
# http://httpd.apache.org/docs-2.0/mod/mod_status.html
SetHandler server-status
这样浏览此 server-status.html 网页时，会输出 apache 的完全状态。使用 server-status.html?refresh=N 参数可以控制刷新频率。输出后的页面如下：
Current Time: Thursday, 16-Dec-2004 04:28:59 CST
Restart Time: Thursday, 16-Dec-2004 04:22:32 CST
Parent Server Generation: 5
Server uptime: 6 minutes 26 seconds
Total accesses: 403 - Total Traffic: 8.9 MB
CPU Usage: u.83 s.05 cu0 cs0 - .228% CPU load
1.04 requests/sec - 23.5 kB/second - 22.5 kB/request
2 requests currently being processed, 198 idle workers
................................................................
................................................................
................................................................
........________________________________________________________
________________K___________________________....................
第七步，编译 apache 时的问题。
首先，找到编译 apache 时候用的 ./configure 语句：
./configure --enable-so --enable-speling --enable-rewrite --enable-ssl --with-ssl=/usr/local/ssl --with-mpm=worker --enable-nonportable-atomics=yes
其次，找到 php 编译时的 ./configure 语句：
./configure \
--enable-force-cgi-redirect \
--enable-fastcgi \
--enable-gd-native-ttf \
--enable-gd-jis-conv \
--enable-bcmath \
--with-apxs2=/usr/local/apache2/bin/apxs \
--with-apache2=/usr/local/apache2 \
--with-mpm=worker \
--with-mysql=/usr/local/mysql \
--with-openssl=/usr/local/ssl \
--with-mcrypt=/usr/local/lib \
--with-gd \
--with-jpeg-dir \
--with-png-dir \
--with-zlib-dir \
--with-freetype-dir
参考文档：
httpd <defunct>
hololi:
assume you are using apache
do you have cpanel or something above apache to manage it ?
anyway, is it the root or a child httpd process ?
when apache starts up it has a root httpd process and spawns child processes to serve requests (these child processes are owned by whomever is user/group in httpd.conf)
it also registers the pid (process ID) of the root process in a file usually called httpd.pid (again name and location can be set in httpd.conf)
the defunct message can be caused by
a) an apache child process went non-linear and its a simple zombie
kill -9 pid should stop this
b) its the root process that dies leaving the children hanging (this is the most common scenario) caused by
1) nonclean shutdown of box (quick power off and no chance to remove httpd.pid)
2) accidental deletion of pid file
best thiing to do is
try and stop apache normally (apachectl stop ? or whatever cpanel etc.. does to stop apache)
ps -ef | grep httpd to make sure there is no httpd processes running
if there are still httpd processes running do a manual kill -9 on each of them
try and find the pid file
i.e find . -name httpd.pid -print
when apache is off this file should NOT exist
if you are sure there are no httpd processes running and you found a pid file simply delete it
I have a script in bash that does a status check on apache and can diagniuse problems such as these if you want
you could write a simple script in bash that searches for defunct processes and kills them if found. This script cound be cron'd etc..
in psudo co
hell I am a bit bored here if you want to pm me I'll knock up a better script for you.. my 30 seconds version looks like ...
#!/bin/bash
echo "running checker script at `date`"
bad_proc=`ps -ef | grep defunct | grep httpd | wc -l `
#bad_proc now contains the number of defunct processes
if [ $bac_proc -gt 0 ]
then # this means it has found defunct httpd processes
# go and kill them
# the killing of them would be something like ..
ps -ef | grep httpd | grep defunct | awk '{print $2}' > /tmp/defunct_processes.dat
# you now have a list of defunct httpd processes in a file called /tmp/defunct_processes.dat
# do a while loop to read these processes and kill them
while read victim; do
echo "attempting a kill -9 on $victim"
kill -9 $victim
done < /tmp/defunct_processes.dat
rm /tmp/defunct_processes.dat
else
echo "no bad processes here"
fi
stdunbar:
Zombie process cannot be killed with any signal. kill -9 does nothing. They are not alive so you can't kill them.
There is a bug in your Apache installation. At the C level something in Apache has not done a wait() correctly.
As was asked, if this is your main httpd then your server is for all practial purposes down. The question then is what spawned the Apache process and why hasn't it reaped the dead child?
Otherwise, if it is a thread or child process then what do you have compiled into Apache to cause this? This is C multi-process programming 101 type of stuff so I'd examine what your modules are doing.
评论