aspnetcore
aspnetcore copied to clipboard
dotnet
Occasional crash in IIS after compile
Description
After updating to .net7 rc1, I am getting occasional crashes of the IIS process after compile has finished.
KernelBase.dll!RaiseException()
coreclr.dll!RaiseTheExceptionInternalOnly(Object * throwable, int rethrow, int) Line 2805
at D:\a\_work\1\s\src\coreclr\vm\excep.cpp(2805)
coreclr.dll!IL_Throw(Object * obj) Line 4025
at D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp(4025)
00007ff7afc5c1d6()
00007ff7af8cc49d()
00007ff7af8cd0f2()
00007ff7ab613b8c()
00007ff7ab613aa7()
00007ff7ac05ad02()
00007ff7ac05ac79()
00007ff7ac077f6d()
00007ff7ac077df6()
00007ff7ac0777e7()
00007ff7ac077723()
00007ff7ac077619()
00007ff7ac07759b()
00007ff7afb2aaa6()
00007ff7afb2a9a5()
00007ff7af8e137a()
00007ff7afc4f5b4()
coreclr.dll!CallDescrWorkerInternal()
coreclr.dll!DispatchCallSimple(unsigned __int64 * pSrc, unsigned long numStackSlotsToCopy, unsigned __int64 pTargetAddress, unsigned long dwDispatchCallSimpleFlags) Line 221
at D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp(221)
coreclr.dll!ThreadNative::KickOffThread_Worker(void * ptr) Line 158
at D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp(158)
[Inline Frame] coreclr.dll!ManagedThreadBase_DispatchInner(ManagedThreadCallState *) Line 7298
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7298)
coreclr.dll!ManagedThreadBase_DispatchMiddle(ManagedThreadCallState * pCallState) Line 7342
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7342)
coreclr.dll!ManagedThreadBase_DispatchOuter(ManagedThreadCallState * pCallState) Line 7501
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7501)
[Inline Frame] coreclr.dll!ManagedThreadBase_FullTransition(void(*)(void *)) Line 7546
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7546)
[Inline Frame] coreclr.dll!ManagedThreadBase::KickOff(void(*)(void *)) Line 7581
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7581)
coreclr.dll!ThreadNative::KickOffThread(void * pass) Line 230
at D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp(230)
kernel32.dll!00007ff9420d7034()
ntdll.dll!00007ff9428a26a1()
I have shadow copy enabled, and this is in the build file, so that iis properly reloads after my build.
<Target Name="PostBuild" AfterTargets="PostBuildEvent">
<Exec Command="del /q $(ProjectDir)app_offline.htm" />
</Target>
<Target Name="PreBuild" BeforeTargets="PreBuildEvent">
<Exec Command="echo "offline" > $(ProjectDir)app_offline.htm" />
</Target>
Reproduction Steps
Not sure
Expected behavior
No crash
Actual behavior
crash
Regression?
It seems to, did not get anything of the same sort in .net6
Known Workarounds
no
Configuration
.net 7 rc1, windows 10 21H2, iis 10.0.19041.1
Other information
no
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
![dotnet-issue-labeler[bot] avatar](https://avatars.githubusercontent.com/in/90031?v=4 "Published by a pub.dev verified publisher"){kind=small}
@davhdavh can you provide a stack with symbols or a full dump? From the partial stack, this looks like a managed exception potentially coming from user code.
This issue has been marked needs-author-action and may be missing some important information.
Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.
Issue Details
Description
After updating to .net7 rc1, I am getting occasional crashes of the IIS process after compile has finished.
KernelBase.dll!RaiseException()
coreclr.dll!RaiseTheExceptionInternalOnly(Object * throwable, int rethrow, int) Line 2805
at D:\a\_work\1\s\src\coreclr\vm\excep.cpp(2805)
coreclr.dll!IL_Throw(Object * obj) Line 4025
at D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp(4025)
00007ff7afc5c1d6()
00007ff7af8cc49d()
00007ff7af8cd0f2()
00007ff7ab613b8c()
00007ff7ab613aa7()
00007ff7ac05ad02()
00007ff7ac05ac79()
00007ff7ac077f6d()
00007ff7ac077df6()
00007ff7ac0777e7()
00007ff7ac077723()
00007ff7ac077619()
00007ff7ac07759b()
00007ff7afb2aaa6()
00007ff7afb2a9a5()
00007ff7af8e137a()
00007ff7afc4f5b4()
coreclr.dll!CallDescrWorkerInternal()
coreclr.dll!DispatchCallSimple(unsigned __int64 * pSrc, unsigned long numStackSlotsToCopy, unsigned __int64 pTargetAddress, unsigned long dwDispatchCallSimpleFlags) Line 221
at D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp(221)
coreclr.dll!ThreadNative::KickOffThread_Worker(void * ptr) Line 158
at D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp(158)
[Inline Frame] coreclr.dll!ManagedThreadBase_DispatchInner(ManagedThreadCallState *) Line 7298
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7298)
coreclr.dll!ManagedThreadBase_DispatchMiddle(ManagedThreadCallState * pCallState) Line 7342
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7342)
coreclr.dll!ManagedThreadBase_DispatchOuter(ManagedThreadCallState * pCallState) Line 7501
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7501)
[Inline Frame] coreclr.dll!ManagedThreadBase_FullTransition(void(*)(void *)) Line 7546
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7546)
[Inline Frame] coreclr.dll!ManagedThreadBase::KickOff(void(*)(void *)) Line 7581
at D:\a\_work\1\s\src\coreclr\vm\threads.cpp(7581)
coreclr.dll!ThreadNative::KickOffThread(void * pass) Line 230
at D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp(230)
kernel32.dll!00007ff9420d7034()
ntdll.dll!00007ff9428a26a1()
I have shadow copy enabled, and this is in the build file, so that iis properly reloads after my build.
<Target Name="PostBuild" AfterTargets="PostBuildEvent">
<Exec Command="del /q $(ProjectDir)app_offline.htm" />
</Target>
<Target Name="PreBuild" BeforeTargets="PreBuildEvent">
<Exec Command="echo "offline" > $(ProjectDir)app_offline.htm" />
</Target>
Reproduction Steps
Not sure
Expected behavior
No crash
Actual behavior
crash
Regression?
It seems to, did not get anything of the same sort in .net6
Known Workarounds
no
Configuration
.net 7 rc1, windows 10 21H2, iis 10.0.19041.1
Other information
no
| Author: | davhdavh |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | - |
As @jeffschwMSFT there is not enough information in this issue to take action. It appears to be a managed exception but no information about the managed callstack was shared. Can you please share either dump or a repro project? If that's not possible, then the managed stack of the crash might be enough. You can get this with the managed debugger, or using SOS in the native debugger.
Putting this in milestone 8.0.0 while we wait for information since this is currently unactionable.
w3wp.zip dump file. sorry it took so long, it only happens once every 2-3 days or so, and sometimes I am too late to attach debugger before IIS forcefully restarts
This dump seems to be at an odd spot. The crash is happening at shutdown.
0:024> ~0s
ntdll!NtDelayExecution+0x14:
00007ff9`428ed684 c3 ret
0:000> k
# Child-SP RetAddr Call Site
00 00000089`ca39f628 00007ff9`404196de ntdll!NtDelayExecution+0x14
01 00000089`ca39f630 00007ff9`3ae8c551 KERNELBASE!SleepEx+0x9e
02 00000089`ca39f6d0 00007ff9`3ae87d97 w3wphost!LISTENER_CHANNEL_LIST_MANAGER::StopAllRunningListenerChannels+0x129
03 00000089`ca39f730 00007ff9`3ae8404d w3wphost!W3WP_HOST::ShutdownProcess+0x8f
.NET Core is not loaded:
0:000> lmvmcoreclr
Browse full module list
start end module name
The stack where the crash is happening indicates that it is happening in an unloaded module.
0:024> .exr -1
ExceptionAddress: 00007ff87132a3b9 (<Unloaded_aspnetcorev2_inprocess.dll>+0x000000000001a3b9)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000008
Parameter[1]: 00007ff87132a3b9
Attempt to execute non-executable address 00007ff87132a3b9
0:024> u 00007ff87132a3b9
<Unloaded_aspnetcorev2_inprocess.dll>+0x1a3b9:
00007ff8`7132a3b9 ?? ???
^ Memory access error in 'u 00007ff87132a3b9'
I noticed that my IIS was using aspnetcorev2 version 16.xxx. despite using .net 7 rc1. I have upgraded to 17.xxx and rc2. https://github.com/dotnet/SqlClient/issues/1418 might be related. When running in .net 6 and <4.1.0/5.0.0 this was causing crashes at similar times, but after upgrade to 4.1.1 I didn't see any crashes until upgrading to .net7 rc1. I have updated to 5.0.1, perhaps they fixed it differently there. Let's see if I keep getting crashes.
Nope still happening. When it happens the "attach debugger" window will popup, but it is no longer possible to attach before it pops up another "attach debugger" prompt. So had to install DebugDiag to get the crash dumps. Here is another 2 dumps w3wp__.NET Core__PID__29112__Date__10_14_2022__Time_01_50_40PM__45__First chance exception 0XE06D7363.zip w3wp__.NET Core__PID__29112__Date__10_14_2022__Time_01_50_40PM__796__First chance exception 0X04242420.zip
'w3wp__.NET Core__PID__29112__Date__10_14_2022__Time_01_50_40PM__796__First chance exception 0X04242420.dmp' This is not an issue. The dump is broken at a exception used for debugger setup. It is benign.
'w3wp__.NET Core__PID__29112__Date__10_14_2022__Time_01_50_40PM__45__First chance exception 0XE06D7363.dmp' This points to an issue in aspnetcorev2_inprocess. I am transferring this issue to aspnetcore.
0:027> .frame 5
05 000000c0`fa67d470 00007ff9`428f16e6 aspnetcorev2_inprocess!__FrameHandler4::CxxCallCatchBlock+0x115 [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 1393]
0:027> dv
pExcept = 0x000000c0`fa67e210
continuationAddresses = void *[2]
TranslatedCatch = 0n0
pForeignException = 0x00000000`00000000
rethrow = 0n0
FrameInfo = struct FrameInfo
HandlerSearchState = 0n9
pSaveException = 0x00000000`00000000
continuationAddress = 0x00000000`00000000
pEstablisherFrame = 0x000000c0`fa67e378
pSaveContext = 0x00000000`00000000
pFrameInfo = 0x000000c0`fa67d4f8
handlerAddress = 0x00007ff8`ca80cb80
FuncInfo = struct FH4::FuncInfo4
pContext = 0x000000c0`fa67edc0
UnwindTryState = 0n8
pThisException = 0x000000c0`fa67f2b0
0:027> .cxr 0x000000c0`fa67edc0
rax=00007ff9403d0000 rbx=00007ff8ca822f88 rcx=0000000000000000
rdx=00007ff900000000 rsi=000000c0fa67f680 rdi=0000000019930520
rip=00007ff940404fd9 rsp=000000c0fa67f4f0 rbp=0000000000000000
r8=0000000000000004 r9=00007ff9403d0000 r10=00007ff942884aff
r11=0000000000000409 r12=000002ab6bb3e1f0 r13=00007ff8ca826038
r14=0000000000000000 r15=00000000ffffffff
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000204
KERNELBASE!RaiseException+0x69:
00007ff9`40404fd9 0f1f440000 nop dword ptr [rax+rax]
0:027> k
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 000000c0`fa67f4f0 00007ff8`ca7ff720 KERNELBASE!RaiseException+0x69
01 000000c0`fa67f5d0 00007ff8`ca7dbf14 aspnetcorev2_inprocess!_CxxThrowException+0x90 [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\throw.cpp @ 75]
02 000000c0`fa67f630 00007ff8`ca7d700b aspnetcorev2_inprocess!std::filesystem::_Throw_fs_error+0x54 [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\include\filesystem @ 1861]
03 (Inline Function) --------`-------- aspnetcorev2_inprocess!std::filesystem::remove_all+0x183 [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\include\filesystem @ 3876]
04 000000c0`fa67f720 00007ff8`ca7d5369 aspnetcorev2_inprocess!<lambda_04a2651c10e466ec62ce4ae77023f039>::operator()+0x37b [D:\a\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\InProcessRequestHandler\inprocessapplication.cpp @ 191]
05 (Inline Function) --------`-------- aspnetcorev2_inprocess!std::invoke+0x30 [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\include\type_traits @ 1558]
06 000000c0`fa67f880 00007ff9`401d1bb2 aspnetcorev2_inprocess!std::thread::_Invoke<std::tuple<<lambda_04a2651c10e466ec62ce4ae77023f039>,std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >,0,1>+0x39 [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.33.31629\include\thread @ 56]
07 000000c0`fa67f8d0 00007ff9`420d7034 ucrtbase!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
08 000000c0`fa67f900 00007ff9`428a26a1 kernel32!BaseThreadInitThunk+0x14
09 000000c0`fa67f930 00000000`00000000 ntdll!RtlUserThreadStart+0x21
0:030> lmv m *aspnetcorev2_inprocess*
Browse full module list
start end module name
00007ff8`ca7d0000 00007ff8`ca830000 aspnetcorev2_inprocess (deferred)
Image path: C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\7.0.0-rc.2.22476.2\aspnetcorev2_inprocess.dll
Image name: aspnetcorev2_inprocess.dll
Browse all global symbols functions data
Timestamp: Mon Sep 26 09:00:37 2022 (6331BE95)
CheckSum: 00063681
ImageSize: 00060000
File version: 17.0.22269.0
Product version: 17.0.22269.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft
ProductName: ASP.NET Core Module Request Handler
InternalName: aspnetcorev2_inprocess.dll
OriginalFilename: aspnetcorev2_inprocess.dll
ProductVersion: 17.0.22269.0
FileVersion: 17.0.22269.0
FileDescription: IIS ASP.NET Core Module V2 Request Handler. Commit: b12b77b241f0a093d53508c3cb2084860bd5339d
LegalCopyright: Copyright (C) Microsoft Corporation
assuming I'm doing this right, it's an access denied trying to delete files in "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files" specifically "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\449"
0:027> dx Debugger.Sessions[0].Processes[29112].Threads[13452].Stack.Frames[7].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[29112].Threads[13452].Stack.Frames[7].SwitchTo()
000000c0`fa67f810 class std::error_code _Ec = class std::error_code
0:027> dx -r1 (*((aspnetcorev2_inprocess!std::error_code *)0xc0fa67f810))
(*((aspnetcorev2_inprocess!std::error_code *)0xc0fa67f810)) [Type: std::error_code]
[+0x000] _Myval : 5 [Type: int]
[+0x008] _Mycat : 0x7ff8ca826038 [Type: std::error_category *]
0:027> !error 5
Error code: (Win32) 0x5 (5) - Access is denied.
0:027> dx Debugger.Sessions[0].Processes[29112].Threads[13452].Stack.Frames[8].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[29112].Threads[13452].Stack.Frames[8].SwitchTo()
000000c0`fa67f880 class IN_PROCESS_APPLICATION::LoadManagedApplication::__l20::<lambda_04a2651c10e466ec62ce4ae77023f039> * this = 0x000002ab`6bbc74e0
@rdi class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * shadowCopyDir = 0x000000c0`fa67f8a0 "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\449"
000000c0`fa67f820 class std::filesystem::path parentDir = class std::filesystem::path
000000c0`fa67f800 class std::filesystem::directory_iterator <begin>$L0 = class std::filesystem::directory_iterator
000000c0`fa67f788 class std::filesystem::directory_iterator <end>$L0 = class std::filesystem::directory_iterator
0:027> dx -r1 (*((aspnetcorev2_inprocess!std::filesystem::path *)0xc0fa67f820))
(*((aspnetcorev2_inprocess!std::filesystem::path *)0xc0fa67f820)) [Type: std::filesystem::path]
preferred_separator : 92 '\' [Type: wchar_t]
[+0x000] _Text : "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files" [Type: std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >]
Why is shadow copy configured to write to C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files? Does your app have full permissions to write/delete from that directory?
Triage: consider improving the error that is thrown when shadow copy folder is absent or inaccessible.
Thanks for contacting us.
We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.
@davhdavh just curious, when it crashes like this do you get anything useful in the event log (not sure if it's system or application event log)? if a managed exception is taking down the process, ideally we'd log a callstack and you wouldn't need a debugger to see it.
Why is shadow copy configured to write to
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files? Does your app have full permissions to write/delete from that directory?
Why not? That folder worked just fine for IIS/.net framework for 15 years, and we are running a framework site with every core site anyway.
Also, there ISN'T a permission problem on that folder.
I can see the shadow folders getting created and cleaned up just fine.
Also:
The subfolders don't have any extra access either, also checked that... 449 is long gone, 463 is my current.
@davhdavh just curious, when it crashes like this do you get anything useful in the event log (not sure if it's system or application event log)? if a managed exception is taking down the process, ideally we'd log a callstack and you wouldn't need a debugger to see it.
Faulting application name: w3wp.exe, version: 10.0.19041.1, time stamp: 0x58c67bf3 Faulting module name: aspnetcorev2_inprocess.dll, version: 17.0.22269.0, time stamp: 0x6331be95
A process serving application pool '.NET Core' suffered a fatal communication error with the Windows Process Activation Service. The process id was '11836'. The data field contains the error number.
Application 'd:\src\cghg\x' was recycled after detecting app_offline.htm.
Exception code: 0xc0000005 Fault offset: 0x00000000000284e4 Faulting process id: 0x2e3c Faulting application start time: 0x01d8dfb7e5cb4643 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\7.0.0-rc.2.22476.2\aspnetcorev2_inprocess.dll Report Id: 83c5534a-008b-432e-aee4-2aedc35d71ef Faulting package full name: Faulting package-relative application ID:
Application: w3wp.exe CoreCLR Version: 7.0.22.47203 .NET Version: 7.0.0-rc.2.22472.3 Description: The process was terminated due to an unhandled exception. Exception Info: exception code c0000005, exception address 00007FF8F3B084E4
Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0
Problem signature: P1: w3wp.exe P2: 10.0.19041.1 P3: 58c67bf3 P4: aspnetcorev2_inprocess.dll P5: 17.0.22269.0 P6: 6331be95 P7: c0000005 P8: 00000000000284e4 P9: P10:
Attached files: \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERA87.tmp.mdmp \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD37.tmp.WERInternalMetadata.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD48.tmp.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD46.tmp.csv \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD76.tmp.txt
These files may be available here:
Analysis symbol: Rechecking for solution: 0 Report Id: 83c5534a-008b-432e-aee4-2aedc35d71ef Report Status: 134217728 Hashed bucket: Cab Guid: 0
Those files no longer exists, I will check next time I get the crash
Why not? That folder worked just fine for IIS/.net framework for 15 years, and we are running a framework site with every core site anyway.
Don't cross the streams. This may not be the reason for the crash but I would avoid pointing 2 independent systems at the same folder. They don't know about each other and they may do things that the other doesn't expect.
It didnt make any difference to use another path... This time the crash dump from WER Uploading crash.zip…