mod_shib crashes apache on child initialization
Description
Environment
Solaris 10 SPARC
Solaris OpenSSL (/usr/sfw/libssl.so)
Solaris Apache 2.0 (/usr/apache2)
curl-7.19.6
log4shib-1.0.4
opensaml-2.3
xerces-c-3.0.1
xmltooling-1.3.3
shibboleth-2.3.1
Activity
Scott Cantor December 17, 2010 at 2:39 PM
Closing after release.
Scott Cantor February 12, 2010 at 6:21 PM
I've also put a patch file on the download site and noted it in a header for that directory.
http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/latest/
Scott Cantor February 12, 2010 at 6:03 PM
http://svn.middleware.georgetown.edu/view/cpp-sp?view=rev&revision=3230
The fix can be applied directly to the configure script inside the package by searching for the affected line, and that should fix it once you reconfigure and rebuild the module.
Thank you for finding this!
Scott Cantor February 12, 2010 at 5:28 PM
Ok, getting 2.2 to build is going to be a mess for me, but in the meantime, the results of the configure step seem to be interesting:
2.3.1:
APXS2_CFLAGS = -O -xarch=386 -xchip=pentium -xspace -Xa -xildoff -xO4 -DSSL_EXPERIMENTAL -DSSL_ENGINE -I/usr/apache2/include -I/usr/sfw/include
2.2.1:
APXS2_CFLAGS = -O -xarch=386 -xchip=pentium -xspace -Xa -xildoff -xO4 -DSSL_EXPERIMENTAL -DSSL_ENGINE -DSOLARIS2=10 -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/apache2/include
I'm willing to bet that's the bug, particularly the defines related to file size, offsets, reentrancy, and the solaris2 thing. It's probably throwing off the size of the request_rec struct members.
So something went haywire in the configure step in 2.3 and I lost a bunch of settings. I love Solaris, really.
For a quick fix, you probably could patch some of those flags from your 2.2 makefile over to the other one and build it with those.
Joel Joel February 12, 2010 at 5:11 PMEdited
yeah, 2.2.1 definately works. Perhaps it's just scribbling on the stack somewhere due to a "lucky" unintialized pointer value?
Updating to either SP 2.3 or 2.3.1 from 2.2.1 causes the SP to fail for any request. Failures happen on virtualhosts that
are not even shib protected. A stack trace shows a crash inside init_request_config:
> gdb /usr/apache2/bin/httpd /var/core/core_XXXX_httpd_80_80_1265743007_7785
...
Program terminated with signal 9, Killed.
...
(gdb) where
#0 0xfe8ec430 in _1cTinit_request_config6FpnLrequest_recpnTshib_request_config_ () from /opt/shibboleth/lib/shibboleth/mod_shib_20.so
#1 0xfe8efeb4 in _1cOshib_post_read6FpnLrequest_reci ()
from /opt/shibboleth/lib/shibboleth/mod_shib_20.so
#2 0x0003e77c in ap_run_post_read_request ()
#3 0x0003d9c0 in ap_read_request ()
#4 0x00026688 in .st_double_foreff ()
#5 0x00038d2c in ap_run_process_connection ()
#6 0x0002d02c in ap_graceful_stop_signalled ()
#7 0x0002d1b8 in ap_graceful_stop_signalled ()
#8 0x0002d230 in ap_graceful_stop_signalled ()
#9 0x0002d648 in ap_mpm_run ()
#10 0x00033e14 in main ()
(gdb)
With log level at debug, I don't see much happening in the error_log besides a shib_child_init prior to the segfault:
[Tue Feb 09 14:24:55 2010] [info] Init: Initializing OpenSSL library
[Tue Feb 09 14:24:55 2010] [info] Init: Seeding PRNG with 136 bytes of entropy
[Tue Feb 09 14:24:55 2010] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Tue Feb 09 14:24:55 2010] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Tue Feb 09 14:24:55 2010] [debug] ssl_scache_dbm.c(406): Inter-Process Session Cache (DBM) Expiry: old: 0, new: 0, removed: 0
[Tue Feb 09 14:24:55 2010] [info] Init: Initializing (virtual) servers for SSL
[Tue Feb 09 14:24:55 2010] [info] mod_ssl/2.0.63 compiled against Server: Apache/2.0.63, Library: OpenSSL/0.9.7d
[Tue Feb 09 14:24:56 2010] [info] Init: Initializing OpenSSL library
[Tue Feb 09 14:24:56 2010] [info] Init: Seeding PRNG with 136 bytes of entropy
[Tue Feb 09 14:24:56 2010] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Tue Feb 09 14:24:57 2010] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Tue Feb 09 14:24:57 2010] [debug] ssl_scache_dbm.c(406): Inter-Process Session Cache (DBM) Expiry: old: 0, new: 0, removed: 0
[Tue Feb 09 14:24:57 2010] [info] Init: Initializing (virtual) servers for SSL
[Tue Feb 09 14:24:57 2010] [info] mod_ssl/2.0.63 compiled against Server: Apache/2.0.63, Library: OpenSSL/0.9.7d
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1290): shib_child_init(8178) starting
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1290): shib_child_init(8179) starting
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1290): shib_child_init(8180) starting
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1290): shib_child_init(8181) starting
[Tue Feb 09 14:24:57 2010] [notice] Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.7d configured – resuming normal operations
[Tue Feb 09 14:24:57 2010] [info] Server built: Jan 18 2009 15:51:05
[Tue Feb 09 14:24:57 2010] [debug] prefork.c(956): AcceptMutex: fcntl (default: fcntl)
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1290): shib_child_init(8182) starting
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1344): shib_child_init() done
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1344): shib_child_init() done
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1344): shib_child_init() done
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1344): shib_child_init() done
[Tue Feb 09 14:24:57 2010] [debug] mod_apache.cpp(1344): shib_child_init() done
[Tue Feb 09 14:25:01 2010] [debug] mod_apache.cpp(1290): shib_child_init(8183) starting
[Tue Feb 09 14:25:01 2010] [debug] mod_apache.cpp(1344): shib_child_init() done
[Tue Feb 09 14:25:04 2010] [notice] child pid 8180 exit signal Segmentation fault (11), possible coredump in /usr/apache2