-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0
-
Fix Version/s: 3.1.0.2
-
Component/s: Error Handling, Web Server Modules
-
Labels:None
-
Environment:
Windows Server 2016 on VMWare
Our website is running mostly Classic ASP, I have set useHeaders="true" useVariables="false"
-
Operating System:Windows
-
CPU Type:x86_64
-
Web Server:IIS 9 (Windows 2016)
A few months ago we started using the Shibboleth Service Provider v3.1.0.1 on IIS on Windows Server 2016 (with all recent Windows updates)
It can be running fine for days but then sometimes w3wp.exe crashes (a few times on a day) with no apparent cause.
Windows Error Reporting shows:
Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a Faulting module name: KERNELBASE.dll, version: 10.0.14393.3659, time stamp: 0x5e9140ed Exception code: 0xe06d7363 Fault offset: 0x0000000000034f38 Faulting process id: 0xc44c Faulting application start time: 0x01d65efa426943f8 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\Windows\System32\KERNELBASE.dll
The 0xe06d7363 translates to EXCEPTION_UNCAUGHT_CXX_EXCEPTION
Next step is to catch a minidump. The Call Stack is shown below
Iis7_shib.dll is very high in the call-stack with IIS7Request::throwError at the top.
It is called by IIS7Request::getRequestBody because a call to m_request->ReadEntityBody fails.
First of all: I am not sure Shibboleth is *causing* the crash, maybe it is just collateral damage. But it may help in finding the root cause
My questions:
- A message goes into the IOException. Is there any way to retrieve that message (some logging perhaps? I could not find it in shibd.log or any other log)
--> In the minidump I can see the error is ERROR_NETNAME_DELETED - ReadEntityBody sound like it is receiving data. Is it fair to generate an (unhandled) exception for ‘just’ a network error?
- If it is only collateral damage: is there a way to catch the exception?
I do see the <InProcess> element supports a (disadvised) catchAll but if I read the sourcecode correctly it may not apply to the IOException
-->Currently testing
(callstack and shibd.log follows)
KERNELBASE.dll!RaiseException-() Unknown [External Code] iis7_shib.dll!IIS7Request::throwError(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & operation, HRESULT hr) Line 452 C++ iis7_shib.dll!IIS7Request::getRequestBody() Line 332 C++ shibsp-lite3_1.dll!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<shibsp::DOMPropertySet>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > >,0> >::_Insert_hint<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > &,std::_Tree_node<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> >,void *> *>(std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > > > > _Where, std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > & _Val, std::_Tree_node<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> >,void *> * _Newnode) Line 1703 C++ shibsp-lite3_1.dll!shibsp::AbstractSPRequest::getHandlerURL(const char * resource) Line 247 C++ xmltooling-lite3_1.dll!xmltooling::TemplateEngine::TemplateParameters::getParameter(const char * name) Line 61 C++ shibsp-lite3_1.dll!shibsp::ServerThread::job() Line 597 C++ xmltooling-lite3_1.dll!xmltooling::TemplateEngine::process(bool visible, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & buf, const char * & lastpos, std::basic_ostream<char,std::char_traits<char> > & os, const xmltooling::TemplateEngine::TemplateParameters & parameters, const std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::basic_string<char,std::char_traits<char>,std::allocator<char> > > & loopentry, const xmltooling::XMLToolingException * e) Line 157 C++ xmltooling-lite3_1.dll!xmltooling::TemplateEngine::run(std::basic_istream<char,std::char_traits<char> > & is, std::basic_ostream<char,std::char_traits<char> > & os, const xmltooling::TemplateEngine::TemplateParameters & parameters, const xmltooling::XMLToolingException * e) Line 286 C++ shibsp-lite3_1.dll!shibsp::ServiceProvider::doHandler(shibsp::SPRequest & request) Line 684 C++ [External Code] shibsp-lite3_1.dll!shibsp::DOMPropertySet::load(const xercesc_3_2::DOMElement * e, log4shib::Category * log, xercesc_3_2::DOMNodeFilter * filter, const shibsp::DOMPropertySet::Remapper * remapper, const xmltooling::QName * unsetter) Line 151 C++ iis7_shib.dll!ShibHttpModule::DoHandler(IHttpContext * pHttpContext, IHttpEventProvider * pProvider) Line 62 C++ [External Code]
Also: this is what shibd.log shows during the crash:
2020-08-19 17:13:17 ERROR Shibboleth.Listener [45]: error reading size of input message 2020-08-19 17:13:17 ERROR Shibboleth.Listener [45]: failed socket call (unknown), result (0): No error 2020-08-19 17:13:17 ERROR Shibboleth.Listener [45]: I/O failure processing request on socket (636) 2020-08-19 17:13:17 ERROR Shibboleth.Listener [47]: error reading size of input message 2020-08-19 17:13:17 ERROR Shibboleth.Listener [47]: failed socket call (unknown), result (0): No error 2020-08-19 17:13:17 ERROR Shibboleth.Listener [47]: I/O failure processing request on socket (440) 2020-08-19 17:13:17 ERROR Shibboleth.Listener [46]: error reading size of input message 2020-08-19 17:13:17 ERROR Shibboleth.Listener [46]: failed socket call (unknown), result (0): No error 2020-08-19 17:13:17 ERROR Shibboleth.Listener [46]: I/O failure processing request on socket (648)
- mentioned in
-
Page Loading...
