IIS App Pool Crash

Description

Our IIS web service keeps crashing with the following error:
A process serving application pool 'app.Production' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2576'. The data field contains the error number.

A debug of the crash shows an access violation that appears to be caused by Shibboleth as referenced from a support request we opened with Microsoft (below):

//From Microsoft PSS

I have finished analyzing the dumps you sent Friday and the crash is being caused by the Shibboleth ISAPI filter. In the crashing call stack below you can see that isapi_shib is calling into our ServerSupportFunction. When I look at the instruction that is causing the AV in our code, I can see that the Shibboleth component is passing in a bad pointer for the ul1 parameter, see http://msdn.microsoft.com/en-us/library/aa503395.aspx.

0:030> kL
Child-SP RetAddr Call Site
00000000`0b23b970 00000001`8000174e filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x174
00000000`0b23bb80 00000001`8000c566 isapi_shib!TerminateFilter+0x45e
00000000`0b23bbc0 00000000`745e6f60 isapi_shib!GetFilterVersion+0x2896
00000000`0b23bc00 00000000`745b3b3c msvcr90!_CallSettingFrame+0x20
00000000`0b23bc30 00000000`77990c21 msvcr90!__CxxCallCatchBlock+0xfc
00000000`0b23bd00 00000001`80007ff2 ntdll!RcFrameConsolidation+0x3
00000000`0b23ea20 000007fe`f67e17e4 isapi_shib!HttpFilterProc+0x2d2
00000000`0b23eec0 000007fe`f67e1e01 filter!W3_FILTER_CONTEXT::NotifyFilters+0x178
00000000`0b23f0e0 000007fe`f8f6a185 filter!GlobalDoWork+0x351
00000000`0b23f310 000007fe`f8f6ab24 iiscore!W3_CONTEXT::SetupStateMachine+0x685
00000000`0b23f820 000007fe`fb4310d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1b0
00000000`0b23f850 000007fe`fb43109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126
00000000`0b23f8b0 000007fe`f8b01fba w3dt!OverlappedCompletionRoutine+0x1c
00000000`0b23f8e0 000007fe`f8b02024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x7a
00000000`0b23f930 000007fe`f8b020a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
00000000`0b23f960 00000000`7783652d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
00000000`0b23f990 00000000`7796c521 kernel32!BaseThreadInitThunk+0xd
00000000`0b23f9c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

As you can see from below the AV is caused by an attempted read from 8000e6f0.

0:030> .exr -1
ExceptionAddress: 000007fef67ebc14 (filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x0000000000000174)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 000000008000e6f0
Attempt to read from address 000000008000e6f0

The memory at the referenced address is free and is Page_Protect

0:030> !address 000000008000e6f0

Usage: Free
Base Address: 00000000`7fff0000
End Address: 00000000`ffb00000
Region Size: 00000000`7fb10000
Type: 00000000
State: 00010000 MEM_FREE
Protect: 00000001 PAGE_NOACCESS

0:030> lmvm isapi_shib
start end module name
00000001`80000000 00000001`80020000 isapi_shib (export symbols) isapi_shib.dll
Loaded symbol image file: isapi_shib.dll
Image path: D:\opt\shibboleth-sp\lib\shibboleth\isapi_shib.dll
Image name: isapi_shib.dll
Timestamp: Sun Jul 03 17:00:27 2011 (4E10D86B)
CheckSum: 00026390
ImageSize: 00020000
File version: 2.4.3.0
Product version: 2.4.3.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: UCAID
ProductName: Shibboleth 2.4.3
InternalName: isapi_shib
OriginalFilename: isapi_shib.dll
ProductVersion: 2, 4, 3, 0
FileVersion: 2, 4, 3, 0
PrivateBuild: 2, 4, 3, 0
SpecialBuild: 2, 4, 3, 0
FileDescription: Shibboleth ISAPI Filter / Extension
LegalCopyright: Copyright © 2011 UCAID
LegalTrademarks: Copyright © 2011 UCAID
Comments: Copyright © 2011 UCAID

Environment

Windows 2008R2 (SP1 + current patches), IIS 7.5 and .NET 3.5 framework

Attachments

2

Activity

Scott Cantor August 7, 2012 at 1:10 AM

No response from reporter to interim fix, but patch was backported into the new release.

Scott Cantor November 2, 2011 at 4:58 PM

Patch ported back to development branch.

http://svn.shibboleth.net/view/cpp-sp?rev=3539&view=rev

Scott Cantor November 2, 2011 at 3:52 PM

Disregard the last, I found the CPU bug, simple accident. Patch is now updated again with the full fix. Will wait for your feedback.

Scott Cantor November 2, 2011 at 3:32 PM

I replaced the patched version with one that only fixes the pointer cast problem. Since I'm not seeing the crash, I can't be sure of that, but the cast matches the new API documentation from MS.

With this change, if the crashing stops, you'll definitely get errors in the browser about the missing variable, but we won't know for sure what's causing that.

Scott Cantor November 2, 2011 at 2:25 PM

No, I saw something similar but it appeared to be unrelated to the SP so I wrote it off at the time. I don't know what could have been caused by any changes I made, but I'll have to dig deeper.

In the meantime, I can back out to a simpler patch that just tries to fix the crash alone.

Fixed

Details

Assignee

Reporter

Original estimate

Components

Fix versions

Affects versions

Created October 28, 2011 at 4:52 PM
Updated August 7, 2012 at 1:11 AM
Resolved August 7, 2012 at 1:10 AM