Check for missing DataSealer during cookie recovery
Basics
Technical
Logistics
Basics
Technical
Logistics
Description
We're migrating our reverse proxy from shibd 2.x to a newer setup, as traffic hits apache (200/300 req/sec) we start to have segmentation fault during request processing.
It seems something related to the session recover, which is not configured and we don't use (we don't have a data sealer configured).
We tried to downgrade from 3.2 to 3.1 then 3.0.4 but we face the same problem in the same function.
We're using apache and changed mpm_event to mpm_worker.
We're using OEL 8.3, which should be binary compatible with RHEL and Centos, I know this is an unsupported configuration.
Here's the traceback of shibd 3.1 (shibboleth-3.1.0-3.1.x86_64.rpm).
Core was generated by `/usr/sbin/shibd -f -F'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fbc08a35716 in shibsp::SSCache::recover (this=0x559f7a02e7e0, app=..., key=0x7fbb8403b910 "_0d5975c944b57a19b5cfd37c3cfe99da", data=0x7fbb8408cb00 "1%3AgwTo%2FTos65aTSxMxw8WGJifjQME8ziyCdJDdlwghpWmYBd%2BUkjKoz4Phg75EBE3AzretN3FWMKmr%0A%2FPw8cEpbEJ2N5lWtbezvMVM6HjBOV7rpqFG7q9G%2FGbyh%2BqTD%2BEOPhPOz2dDcxmwFOtl1BMZNaMyV%0AdyGUpZHuWmqqDx%2BRzTnlx%2F"...) at impl/StorageServiceSessionCache.cpp:1176 1176 unwrapped = XMLToolingConfig::getConfig().getDataSealer()->unwrap(dup); [Current thread is 1 (Thread 0x7fbb7a7fc700 (LWP 19469))] Missing separate debuginfos, use: yum debuginfo-install brotli-1.0.6-2.el8.x86_64 cyrus-sasl-lib-2.1.27-5.el8.x86_64 glibc-2.28-127.0.3.el8_3.2.x86_64 keyutils-libs-1.5.10-6.el8.x86_64 krb5-libs-1.18.2-5.el8.x86_64 libblkid-2.32.1-24.el8.x86_64 libcap-2.26-4.el8.x86_64 libcom_err-1.45.6-1.el8.x86_64 libcurl-7.61.1-14.el8_3.1.x86_64 libgcc-8.3.1-5.1.0.2.el8.x86_64 libgcrypt-1.8.5-4.el8.x86_64 libgpg-error-1.31-1.el8.x86_64 libidn2-2.2.0-1.el8.x86_64 liblog4shib2-2.0.0-4.1.x86_64 libmount-2.32.1-24.el8.x86_64 libnghttp2-1.33.0-3.el8_2.1.x86_64 libpsl-0.20.2-6.el8.x86_64 libsaml11-3.1.0-3.2.x86_64 libselinux-2.9-4.el8_3.x86_64 libssh-0.9.4-2.el8.x86_64 libunistring-0.9.9-3.el8.x86_64 libuuid-2.32.1-24.el8.x86_64 libxerces-c-3_2-3.2.3-3.2.x86_64 libxml-security-c20-2.0.2-4.1.x86_64 libxmltooling9-3.1.0-5.2.x86_64 lz4-libs-1.8.3-2.el8.x86_64 openldap-2.4.46-15.el8.x86_64 openssl-libs-1.1.1g-15.el8_3.x86_64 pcre2-10.32-2.el8.x86_64 systemd-libs-239-41.0.1.el8_3.2.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.2.el8_3.x86_64 (gdb) bt #0 0x00007fbc08a35716 in shibsp::SSCache::recover (this=0x559f7a02e7e0, app=..., key=0x7fbb8403b910 "_0d5975c944b57a19b5cfd37c3cfe99da", data=0x7fbb8408cb00 "1%3AgwTo%2FTos65aTSxMxw8WGJifjQME8ziyCdJDdlwghpWmYBd%2BUkjKoz4Phg75EBE3AzretN3FWMKmr%0A%2FPw8cEpbEJ2N5lWtbezvMVM6HjBOV7rpqFG7q9G%2FGbyh%2BqTD%2BEOPhPOz2dDcxmwFOtl1BMZNaMyV%0AdyGUpZHuWmqqDx%2BRzTnlx%2F"...) at impl/StorageServiceSessionCache.cpp:1176 #1 0x00007fbc08a397c2 in shibsp::SSCache::receive (this=0x559f7a02e7e0, in=..., out=...) at /usr/include/c++/8/bits/char_traits.h:287 #2 0x00007fbc08a6db52 in shibsp::ListenerService::receive (this=0x559f7a149b80, in=..., out=...) at remoting/impl/ListenerService.cpp:151 #3 0x00007fbc08a72052 in shibsp::ServerThread::job (this=0x559f7a7b43f0) at remoting/impl/SocketListener.cpp:563 #4 0x00007fbc08a72c1a in shibsp::ServerThread::run (this=0x559f7a7b43f0) at remoting/impl/SocketListener.cpp:497 #5 0x00007fbc08a72dd5 in server_thread_fn (arg=0x559f7a7b43f0) at remoting/impl/SocketListener.cpp:433 #6 0x00007fbc05ca416a in start_thread () from /usr/lib64/libpthread.so.0 #7 0x00007fbc059d5f83 in clone () from /usr/lib64/libc.so.6
Environment
OEL 8.3
httpd-2.4.37-30
shibboleth 3.1.0-3.1
We have 3 application overrides (the default application is "empty") and we're using the native mapper (we tried shibrequestsetting in location / before).
Activity
Scott Cantor April 22, 2021 at 8:03 PM
Committed fix to check for component before recovery attempt.
Scott Cantor April 22, 2021 at 8:01 PM
If the old SP were 2.x, there woudn't be a recovery cookie. You have something confgured or that had been configured to drop that cookie. At least in this specific instance.
Unidentified Legacy Account April 22, 2021 at 5:26 PM
Edited
Thank you very much, I'll try to configure a DataSealer.
The clients are coming with existing sessions/cookies from the old sp, we're redirecting them to the new server via a DNS change so we have a lot of "invalid" session when they arrive.
The old sp is a 2.6.1.
Scott Cantor April 22, 2021 at 4:02 PM
A workaround for the crash of course should be to just install a DataSealer even if it's not meant to be used, it won't do anything really.
If you can let me know how you want to be credited on the advisory, I'll probably get an update done by sometime next week.
Scott Cantor April 22, 2021 at 3:48 PM
In your specific case, sorry, but you have a cookie there. Why that is, I couldn't say, but the SP put it there or your client(s) did, one or the other. That's why it's attempting the recovery. Not having it will fix that particular crash, but not prevent it from being possible.
We're migrating our reverse proxy from shibd 2.x to a newer setup, as traffic hits apache (200/300 req/sec) we start to have segmentation fault during request processing.
It seems something related to the session recover, which is not configured and we don't use (we don't have a data sealer configured).
We tried to downgrade from 3.2 to 3.1 then 3.0.4 but we face the same problem in the same function.
We're using apache and changed mpm_event to mpm_worker.
We're using OEL 8.3, which should be binary compatible with RHEL and Centos, I know this is an unsupported configuration.
Here's the traceback of shibd 3.1 (shibboleth-3.1.0-3.1.x86_64.rpm).
Core was generated by `/usr/sbin/shibd -f -F'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fbc08a35716 in shibsp::SSCache::recover (this=0x559f7a02e7e0, app=...,
key=0x7fbb8403b910 "_0d5975c944b57a19b5cfd37c3cfe99da",
data=0x7fbb8408cb00 "1%3AgwTo%2FTos65aTSxMxw8WGJifjQME8ziyCdJDdlwghpWmYBd%2BUkjKoz4Phg75EBE3AzretN3FWMKmr%0A%2FPw8cEpbEJ2N5lWtbezvMVM6HjBOV7rpqFG7q9G%2FGbyh%2BqTD%2BEOPhPOz2dDcxmwFOtl1BMZNaMyV%0AdyGUpZHuWmqqDx%2BRzTnlx%2F"...)
at impl/StorageServiceSessionCache.cpp:1176
1176 unwrapped = XMLToolingConfig::getConfig().getDataSealer()->unwrap(dup);
[Current thread is 1 (Thread 0x7fbb7a7fc700 (LWP 19469))]
Missing separate debuginfos, use: yum debuginfo-install brotli-1.0.6-2.el8.x86_64 cyrus-sasl-lib-2.1.27-5.el8.x86_64 glibc-2.28-127.0.3.el8_3.2.x86_64 keyutils-libs-1.5.10-6.el8.x86_64 krb5-libs-1.18.2-5.el8.x86_64 libblkid-2.32.1-24.el8.x86_64 libcap-2.26-4.el8.x86_64 libcom_err-1.45.6-1.el8.x86_64 libcurl-7.61.1-14.el8_3.1.x86_64 libgcc-8.3.1-5.1.0.2.el8.x86_64 libgcrypt-1.8.5-4.el8.x86_64 libgpg-error-1.31-1.el8.x86_64 libidn2-2.2.0-1.el8.x86_64 liblog4shib2-2.0.0-4.1.x86_64 libmount-2.32.1-24.el8.x86_64 libnghttp2-1.33.0-3.el8_2.1.x86_64 libpsl-0.20.2-6.el8.x86_64 libsaml11-3.1.0-3.2.x86_64 libselinux-2.9-4.el8_3.x86_64 libssh-0.9.4-2.el8.x86_64 libunistring-0.9.9-3.el8.x86_64 libuuid-2.32.1-24.el8.x86_64 libxerces-c-3_2-3.2.3-3.2.x86_64 libxml-security-c20-2.0.2-4.1.x86_64 libxmltooling9-3.1.0-5.2.x86_64 lz4-libs-1.8.3-2.el8.x86_64 openldap-2.4.46-15.el8.x86_64 openssl-libs-1.1.1g-15.el8_3.x86_64 pcre2-10.32-2.el8.x86_64 systemd-libs-239-41.0.1.el8_3.2.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.2.el8_3.x86_64
(gdb) bt
#0 0x00007fbc08a35716 in shibsp::SSCache::recover (this=0x559f7a02e7e0, app=...,
key=0x7fbb8403b910 "_0d5975c944b57a19b5cfd37c3cfe99da",
data=0x7fbb8408cb00 "1%3AgwTo%2FTos65aTSxMxw8WGJifjQME8ziyCdJDdlwghpWmYBd%2BUkjKoz4Phg75EBE3AzretN3FWMKmr%0A%2FPw8cEpbEJ2N5lWtbezvMVM6HjBOV7rpqFG7q9G%2FGbyh%2BqTD%2BEOPhPOz2dDcxmwFOtl1BMZNaMyV%0AdyGUpZHuWmqqDx%2BRzTnlx%2F"...)
at impl/StorageServiceSessionCache.cpp:1176
#1 0x00007fbc08a397c2 in shibsp::SSCache::receive (this=0x559f7a02e7e0, in=..., out=...)
at /usr/include/c++/8/bits/char_traits.h:287
#2 0x00007fbc08a6db52 in shibsp::ListenerService::receive (this=0x559f7a149b80, in=..., out=...)
at remoting/impl/ListenerService.cpp:151
#3 0x00007fbc08a72052 in shibsp::ServerThread::job (this=0x559f7a7b43f0) at remoting/impl/SocketListener.cpp:563
#4 0x00007fbc08a72c1a in shibsp::ServerThread::run (this=0x559f7a7b43f0) at remoting/impl/SocketListener.cpp:497
#5 0x00007fbc08a72dd5 in server_thread_fn (arg=0x559f7a7b43f0) at remoting/impl/SocketListener.cpp:433
#6 0x00007fbc05ca416a in start_thread () from /usr/lib64/libpthread.so.0
#7 0x00007fbc059d5f83 in clone () from /usr/lib64/libc.so.6