perfview icon indicating copy to clipboard operation
perfview copied to clipboard

WPA sometimes fails to load perf data

Open ami-GS opened this issue 2 years ago • 6 comments

Please refer WPA perf extension (PerfDataExtensions). original issue link is blow

https://github.com/microsoft/Microsoft-Performance-Tools-Linux-Android/issues/107

ami-GS avatar Jan 28 '23 00:01 ami-GS

Can you please share more data about the error and what you're see?

brianrob avatar Jan 28 '23 00:01 brianrob

hmm, nothing other than these. the Loading dialogue quickly disappears and Diagnostic Console shows the error. I feel this is 50% probability Or maybe 100% failed at first, then second loading succeeds

ami-GS avatar Jan 28 '23 00:01 ami-GS

I see - this is an issue with the ParallelLinuxPerfScriptStackSource:

Unable to process 'file:///C:/Users/daikiaminaka/Downloads/handshake_f1000a.perf.data.txt': System.AggregateException: One or more errors occurred. (Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.) (Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.) (Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.) (Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.)
 ---> System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptProcessNameBuilder.SaveProcessName(StackSourceFrameIndex frameIndex, String processName, Int32 processId)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.CreateSampleFor(LinuxEvent linuxEvent, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.ParallelLinuxPerfScriptStackSource.<>c__DisplayClass1_1.<DoInterning>b__0(Object givenArrayIndex)
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAllCore(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.Diagnostics.Tracing.StackSources.ParallelLinuxPerfScriptStackSource.DoInterning()
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternAllLinuxEvents(Stream stream)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource..ctor(String path, Boolean doThreadTime)
   at Microsoft.Diagnostics.Tracing.StackSources.ParallelLinuxPerfScriptStackSource..ctor(String path, Boolean doThreadTime)
   at PerfDataProcessingSource.PerfDataCustomDataProcessor.ProcessAsyncCore(IProgress`1 progress, CancellationToken cancellationToken) in C:\Users\daikiaminaka\workspace\Microsoft-Performance-Tools-Linux-Android\PerfDataExtensions\SourceDataCookers\PerfDataCustomDataProcessor.cs:line 80
   at Microsoft.Performance.SDK.Processing.CustomDataProcessor.ProcessAsync(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Runtime.ProcessingSourceExecutor.ExecuteAsync(CancellationToken cancellationToken)
 ---> (Inner Exception #1) System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptProcessNameBuilder.SaveProcessName(StackSourceFrameIndex frameIndex, String processName, Int32 processId)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.CreateSampleFor(LinuxEvent linuxEvent, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.ParallelLinuxPerfScriptStackSource.<>c__DisplayClass1_1.<DoInterning>b__0(Object givenArrayIndex)
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)<---

 ---> (Inner Exception #2) System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptProcessNameBuilder.SaveProcessName(StackSourceFrameIndex frameIndex, String processName, Int32 processId)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.CreateSampleFor(LinuxEvent linuxEvent, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.ParallelLinuxPerfScriptStackSource.<>c__DisplayClass1_1.<DoInterning>b__0(Object givenArrayIndex)
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)<---

 ---> (Inner Exception #3) System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptProcessNameBuilder.SaveProcessName(StackSourceFrameIndex frameIndex, String processName, Int32 processId)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.InternFrames(IEnumerator`1 frameIterator, StackSourceCallStackIndex stackIndex, Int32 processID, Nullable`1 threadid, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.LinuxPerfScriptStackSource.CreateSampleFor(LinuxEvent linuxEvent, BlockedTimeAnalyzer blockedTimeAnalyzer)
   at Microsoft.Diagnostics.Tracing.StackSources.ParallelLinuxPerfScriptStackSource.<>c__DisplayClass1_1.<DoInterning>b__0(Object givenArrayIndex)
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)<---

brianrob avatar Jan 28 '23 00:01 brianrob

This looks like it's a missing lock or other multi-threaded access protection mechanism. This will require a change in TraceEvent to fix.

brianrob avatar Jan 28 '23 00:01 brianrob

Is it PerfView side or Extension side?

ami-GS avatar Jan 28 '23 00:01 ami-GS

This is an issue in TraceEvent which is the library that drives PerfView. This is the right repo for this issue.

brianrob avatar Jan 28 '23 00:01 brianrob