[H-GEN] Apache segfaulting... how to debug

[Joel Michael]

[ Humbug *General* list - semi-serious discussions about Humbug and     ]
[ Unix-related topics. Posts from non-subscribed addresses will vanish. ]

Michael Anthon wrote:

> I am not getting any reports of problems from the users of the system 
> so I'm not currently sure if this is actually causing a problem.  It's 
> possible that it is and that the users are just reloading the affected 
> page.  The actual cause of the segfault is also eluding me.  I can't 
> find anything in particular that triggers it.  It seems to be random 
> but relatively regular, on the order of 1% of requests judging by the 
> logs since Sunday night.  This is not a new problem, I've noticed it 
> before and put it into the "I'll fix that later cause nobody has 
> noticed it" basket.
>
> What I would like to do is to track down the cause of these faults 
> (obviously) so that I can fix it or ignore it.  I figured the best 
> place to start was to get a core dump from apache and backtrace it... 
> HAH ! I'm stuffed if I can get apache to give me a core file.

Yeah, Apache generally won't give you a core file.  One thing you can 
do, however, is strace a single instance of httpd, and wait.

Try running:

strace /usr/sbin/httpd -X (+ any other options you give httpd)

(from memory, it's been a while since I've had to do this.  It may be a 
lower-case -x that you need to provide)

and wait until it crashes.  This prevents apache from forking, only 
allowing apache to serve a single request at a time, so only do this on 
a non mission critical server!

On another tangent, some things that I know will cause a segfault in 
apache are (as Greg suggested) hardware errors, and strange interactions 
between apache modules.  I know, first hand, that Apache will segfault, 
if it was compiled straight from source, mod_perl or mod_php is loaded, 
and you try to use XML functions that were compiled using the system's 
XML libraries, as Apache is (no longer?) statically compiled with an 
older version of the XML libraries.  Another good one is a bizarre 
interaction between the Informix mod_explode and mod_perl, where apache 
will just fall over for no apparent reason on one machine, but be fine 
on an identical machine (hardware- and software-wise).

(AAAAAAAGGGGGHHHHHH!!! I thought I buried those memories!)

So, if you are 110% sure that your hardware is fine (memtest86 is a good 
one to run), remove any apache modules you don't use, and any php 
modules you don't use.  If possible, set up a test case, even if it 
means copying your entire site and config to another server, and 
replaying the access logs against the second server.

It could also be some Apache exploit/DoS running around that is hitting 
you, examine your access logs very closely.

Anyway, hope this helps steer you in the direction of success (or, at 
least, less annoyance).

-- 
Joel Michael
Seeking employment...  Anoyone need a RHCE (+Red Hat Enterprise) at the 
moment?


--
* This is list (humbug) general handled by majordomo at lists.humbug.org.au .
* Postings to this list are only accepted from subscribed addresses of
* lists 'general' or 'general-post'.  See http://www.humbug.org.au/