fail fast c0000409 exceptions while logging telemetry via background tasks

[sulahane]

Describe your environment. We are using win10-cs solution of 1DS in our UWP app. We are getting high volume of fast fail c0000409 exceptions from 1DS while we are trying to log telemetry via UWP background tasks.

Steps to reproduce. Build win10-cs solution and added it as nuget in UWP app. Foreground tasks are working without any issue, but high volume of exceptions are logged by background tasks.

What is the actual behavior? Multiple exceptions seen on Watson with slightly different flavors

1. fail_fast_invalid_arg_c0000409_microsoft.applications.telemetry.windows.dll! microsoft :: applications :: events :: activeloggercall :: activeloggercall
2. fail_fast_fatal_app_exit_c0000409_microsoft.applications.telemetry.windows.dll! microsoft :: applications :: events :: activeloggercall :: activeloggercall
3. fail_fast_guard_icall_check_failure_c0000409_microsoft.applications.telemetry.windows.dll! microsoft :: applications :: events :: activeloggercall :: activeloggercall
4. invalid_pointer_read_fail_fast_guard_icall_check_failure_c0000409_microsoft.applications.telemetry.windows.dll! microsoft :: applications :: events :: activeloggercall :: activeloggercall

Additional context. Stack trace 1

Frame	Image	Function	Offset
0	Microsoft.Applications.Telemetry.Windows.dll	Microsoft::Applications::Events::ActiveLoggerCall::ActiveLoggerCall	0x000000000000001A
1	Microsoft.Applications.Telemetry.Windows.dll	Microsoft::Applications::Events::Logger::LogEvent	0x000000000000003E
2	Microsoft.Applications.Telemetry.Windows.dll	Microsoft::Applications::Telemetry::Windows::Logger::[Microsoft::Applications::Telemetry::Windows::ILogger]::LogEvent	0x0000000000000119
3	Microsoft.Applications.Telemetry.Windows.dll	Microsoft::Applications::Telemetry::Windows::Logger::[Microsoft::Applications::Telemetry::Windows::ILogger]::__abi_Microsoft_Applications_Telemetry_Windows_ILogger____abi___LogEvent__1	0x000000000000000D

Stack trace 2

LogConfiguration is as follows:

LogManager.SetTransmitProfile(TransmitProfile.RealTime); LogConfiguration config = new LogConfiguration() { MaxTeardownUploadTimeInSec = 5, MinTraceLevel = ACTTraceLevel.ACTTraceLevel_Trace, TraceLevelMask = 0xFFFFFFFF, // API calls & Global mask for general messages SdkMode = SdkModeTypes.SdkModeTypes_CS, // Direct upload rather than through UTC };

[lalitb]

@sulahane - Do you plan to debug this further as you have reproduction of the crash? It is difficult to get the root cause from the stack trace shared. Also, I don't think any of the active contributors/maintainers are using the UWP app. These fail-fast exceptions are mostly related to memory leak/stack-corruption somewhere in code (not necessarily in the 1ds), it may just happen to fail inside 1ds while uploading events.

The 1ds sdk stack trace LogEvent -> ActiveLoggerCall is called during the hot-path of event upload across all the platforms and any such memory-leak/stack-corruption would have been easily seen there.

ps - I have removed the P1 tag, I don't think it is an important bug to fix from 1ds sdk prospective, unless it is reported by multiple users across different platforms.

[sulahane]

@lalitb Yes, I am planning to debug this further. Are there any tools that can be used to trace the memory leaks/stack corruption? when we switch from managed to unmanaged code?

[lalitb]

@sulahane - We don't use any memory/cpu profiling tools as part of CI test to catch memory leaks., but you can use something like google-sanitisers. ASAN ( from google-sanitisers) and valgrind are popular memory profiling tools for unmanaged code.

[sulahane]

This issue is result of "use after free" in UWP app. Proposed solution in https://github.com/microsoft/cpp_client_telemetry/issues/822 can resolve this issue.