flutter icon indicating copy to clipboard operation
flutter copied to clipboard

Published 20 hours ago • verified publisher icon flutter

mac/ios devicelab builds flaking on observatory port time out

Open christopherfujino opened this issue 3 years ago • 25 comments

https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20complex_layout_ios__start_up/5308/overview

Started on engine roll: https://github.com/flutter/flutter/commit/09c5863b9f7959c4ff8b5cf303fba2c867b79335 (PR https://github.com/flutter/flutter/pull/106744) which only contains two Skia rolls, probably unrelated.

I don't think this is the same as https://github.com/flutter/flutter/issues/103511 (but I could be wrong), because:

  1. It passed the ios-debug-symbol-doctor diagnose step, and flutter doctor does not report any ios device errors
  2. The lldb backtrace looks different:
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9caac libsystem_kernel.dylib`mach_msg_trap + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] libsystem_kernel.dylib`mach_msg_trap:
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] ->  0x1bac9caac <+8>: ret    
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] libsystem_kernel.dylib`mach_msg_overwrite_trap:
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     0x1bac9cab0 <+0>: mov    x16, #-0x20
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     0x1bac9cab4 <+4>: svc    #0x80
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     0x1bac9cab8 <+8>: ret    
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] Target 0: (Runner) stopped.
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] thread backtrace all
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] process continue
[complex_layout_ios__start_up] [STDOUT] stdout: [  +41 ms] * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   * frame #0: 0x00000001bac9caac libsystem_kernel.dylib`mach_msg_trap + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001bac9d07c libsystem_kernel.dylib`mach_msg + 72
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x00000001803cfd78 CoreFoundation`__CFRunLoopServiceMachPort + 368
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x00000001803d4080 CoreFoundation`__CFRunLoopRun + 1160
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001803e7240 CoreFoundation`CFRunLoopRunSpecific + 572
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #5: 0x00000001a0eb7988 GraphicsServices`GSEventRunModal + 160
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #6: 0x0000000182be741c UIKitCore`-[UIApplication _run] + 1080
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #7: 0x0000000182980b88 UIKitCore`UIApplicationMain + 336
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #8: 0x000000010415bd3c Runner`main(argc=<unavailable>, argv=<unavailable>) at main.m:10:16 [opt]
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #9: 0x00000001042d43d0 dyld`start + 444
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #4, name = 'com.apple.uikit.eventfetch-thread'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9caac libsystem_kernel.dylib`mach_msg_trap + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001bac9d07c libsystem_kernel.dylib`mach_msg + 72
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x00000001803cfd78 CoreFoundation`__CFRunLoopServiceMachPort + 368
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x00000001803d4080 CoreFoundation`__CFRunLoopRun + 1160
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001803e7240 CoreFoundation`CFRunLoopRunSpecific + 572
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #5: 0x0000000181af4efc Foundation`-[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 232
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #6: 0x0000000181b34010 Foundation`-[NSRunLoop(NSRunLoop) runUntilDate:] + 88
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #7: 0x0000000182b6686c UIKitCore`-[UIEventFetcher threadMain] + 512
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #8: 0x0000000181b41c0c Foundation`__NSThread__start__ + 792
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #9: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #5
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9d014 libsystem_kernel.dylib`__workq_kernreturn + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #10, name = 'io.flutter.1.ui'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9caac libsystem_kernel.dylib`mach_msg_trap + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001bac9d07c libsystem_kernel.dylib`mach_msg + 72
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x00000001803cfd78 CoreFoundation`__CFRunLoopServiceMachPort + 368
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x00000001803d4080 CoreFoundation`__CFRunLoopRun + 1160
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001803e7240 CoreFoundation`CFRunLoopRunSpecific + 572
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #5: 0x0000000105a51b9c Flutter`fml::MessageLoopDarwin::Run() + 88
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #6: 0x0000000105a50c1c Flutter`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::Thread::Thread(std::__1::function<void (fml::Thread::ThreadConfig const&)> const&, fml::Thread::ThreadConfig const&)::$_0> >(void*) + 208
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #7: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #11, name = 'io.flutter.1.raster'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9caac libsystem_kernel.dylib`mach_msg_trap + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001bac9d07c libsystem_kernel.dylib`mach_msg + 72
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x00000001803cfd78 CoreFoundation`__CFRunLoopServiceMachPort + 368
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x00000001803d4080 CoreFoundation`__CFRunLoopRun + 1160
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001803e7240 CoreFoundation`CFRunLoopRunSpecific + 572
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #5: 0x0000000105a51b9c Flutter`fml::MessageLoopDarwin::Run() + 88
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #6: 0x0000000105a50c1c Flutter`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::Thread::Thread(std::__1::function<void (fml::Thread::ThreadConfig const&)> const&, fml::Thread::ThreadConfig const&)::$_0> >(void*) + 208
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #7: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #12, name = 'io.flutter.1.io'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9caac libsystem_kernel.dylib`mach_msg_trap + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001bac9d07c libsystem_kernel.dylib`mach_msg + 72
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x00000001803cfd78 CoreFoundation`__CFRunLoopServiceMachPort + 368
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x00000001803d4080 CoreFoundation`__CFRunLoopRun + 1160
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001803e7240 CoreFoundation`CFRunLoopRunSpecific + 572
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #5: 0x0000000105a51b9c Flutter`fml::MessageLoopDarwin::Run() + 88
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #6: 0x0000000105a50c1c Flutter`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::Thread::Thread(std::__1::function<void (fml::Thread::ThreadConfig const&)> const&, fml::Thread::ThreadConfig const&)::$_0> >(void*) + 208
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #7: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #13, name = 'io.flutter.1.profiler'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9caac libsystem_kernel.dylib`mach_msg_trap + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001bac9d07c libsystem_kernel.dylib`mach_msg + 72
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x00000001803cfd78 CoreFoundation`__CFRunLoopServiceMachPort + 368
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x00000001803d4080 CoreFoundation`__CFRunLoopRun + 1160
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001803e7240 CoreFoundation`CFRunLoopRunSpecific + 572
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #5: 0x0000000105a51b9c Flutter`fml::MessageLoopDarwin::Run() + 88
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #6: 0x0000000105a50c1c Flutter`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::Thread::Thread(std::__1::function<void (fml::Thread::ThreadConfig const&)> const&, fml::Thread::ThreadConfig const&)::$_0> >(void*) + 208
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #7: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #14, name = 'io.worker.1'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9d484 libsystem_kernel.dylib`__psynch_cvwait + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001db7dbbd4 libsystem_pthread.dylib`_pthread_cond_wait$VARIANT$mp + 1240
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x0000000197ce48e0 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 24
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x0000000105a4b8b0 Flutter`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::ConcurrentMessageLoop::ConcurrentMessageLoop(unsigned long)::$_0> >(void*) + 304
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #15, name = 'io.worker.2'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9d484 libsystem_kernel.dylib`__psynch_cvwait + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x00000001db7dbbd4 libsystem_pthread.dylib`_pthread_cond_wait$VARIANT$mp + 1240
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x0000000197ce48e0 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 24
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x0000000105a4b8b0 Flutter`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, fml::ConcurrentMessageLoop::ConcurrentMessageLoop(unsigned long)::$_0> >(void*) + 304
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #4: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]   thread #16, name = 'dart:io EventHandler'
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #0: 0x00000001bac9e294 libsystem_kernel.dylib`kevent + 8
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #1: 0x0000000105c34710 Flutter`dart::bin::EventHandlerImplementation::EventHandlerEntry(unsigned long) + 448
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #2: 0x0000000105c66044 Flutter`dart::bin::ThreadStart(void*) + 44
[complex_layout_ios__start_up] [STDOUT] stdout: [        ]     frame #3: 0x00000001db7d5348 libsystem_pthread.dylib`_pthread_start + 116
[complex_layout_ios__start_up] [STDOUT] stdout: [        ] (lldb) Process 4032 resuming

christopherfujino avatar Jun 28 '22 19:06 christopherfujino

Also happening on Mac_ios integration_ui_ios_textfield (on mac-28 like the other one) so I suspect it's bot-related rather than test:

https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20integration_ui_ios_textfield/4405/overview

christopherfujino avatar Jun 28 '22 19:06 christopherfujino

Here are four consecutive builds from mac-28:

  1. success: https://chromium-swarm.appspot.com/task?id=5bc45344788bb910
  2. failure: https://chromium-swarm.appspot.com/task?id=5bc4583d62a43e10
  3. failure: https://chromium-swarm.appspot.com/task?id=5bc4793594903e10
  4. success: https://chromium-swarm.appspot.com/task?id=5bc49c8ef1ce8910

christopherfujino avatar Jun 28 '22 21:06 christopherfujino

Here are some failed builds from other bots: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20ios_defines_test/5373/overview (mac30) https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20flutter_gallery_ios__start_up/5276/overview (mac27) https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5127/overview (mac6) https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5086/overview (mac8)

keyonghan avatar Jun 28 '22 21:06 keyonghan

Here are some failed builds from other bots: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20ios_defines_test/5373/overview (mac30) https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20flutter_gallery_ios__start_up/5276/overview (mac27) https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5127/overview (mac6) https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5086/overview (mac8)

At least in https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20ios_defines_test/5373/overview, I see in the doctor there were actually un-sync'd debug symbols in the doctor run, while the ios-debug-symbol-doctor diagnose step was green. This means either: the state was repaired in between the flutter doctor and ios-debug-symbol-doctor runs, or there's a bug in the latter. Going to add more logging to ios-debug-symbol-doctor so we have more data.

christopherfujino avatar Jun 28 '22 22:06 christopherfujino

tree currently red on this: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5164/overview on mac-25

christopherfujino avatar Jun 28 '22 22:06 christopherfujino

https://github.com/flutter/cocoon/pull/1948 adds debug logging which will hopefully help diagnose this

christopherfujino avatar Jun 28 '22 22:06 christopherfujino

Another instance, however this time no sign of the debug symbol message in flutter doctor or the ios-debug-symbol-doctor: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20flutter_view_ios__start_up/4535/overview

christopherfujino avatar Jul 01 '22 21:07 christopherfujino

From the logs I don't see anything suspicious. I am adding this to the infra ticket queue as I believe someone will need to look at the phone while it is in this state.

christopherfujino avatar Jul 06 '22 20:07 christopherfujino

FYI @hellohuanlin framework gardener this week

christopherfujino avatar Jul 06 '22 20:07 christopherfujino

@christopherfujino do you need anything specific from the bots when they are in this state?

yusuf-goog avatar Jul 07 '22 18:07 yusuf-goog

@christopherfujino do you need anything specific from the bots when they are in this state?

Well, ultimately I need to know why they're stuck :) Hopefully there's some clue on the device, like a bulging battery or dialog that isn't dismissed?

christopherfujino avatar Jul 07 '22 18:07 christopherfujino

Speaking with @christopherfujino on debugging this in person in the lab.

yusuf-goog avatar Jul 11 '22 17:07 yusuf-goog

@yusuf-goog were you able to find anything wrong with the bots/phones?

christopherfujino avatar Jul 12 '22 23:07 christopherfujino

Testbeds that hit this issue looked okay last Friday when I visited the lab, but didn't see this issue the whole day.

This seems still happening frequently across multiple tests: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5438/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5431/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20flutter_view_ios__start_up/4766/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20flutter_gallery_ios__start_up/5584/overview

keyonghan avatar Jul 19 '22 16:07 keyonghan

No update on this. I am removing my assignment as I am not sure how to investigate further, I don't believe it is related to https://github.com/flutter/flutter/issues/103511.

christopherfujino avatar Jul 19 '22 23:07 christopherfujino

@zanderso Could you help take a look? (Assigning to @zanderso based on test ownership)

keyonghan avatar Jul 19 '22 23:07 keyonghan

The tool is passing --disable-service-auth-codes unconditionally here.

Which both gets passed along to the Dart VM here, and is used to set up the engine's service isolate here.

Which should disable auth codes here.

However, from examining the device logs, it looks like these tests are hanging because the VM service URIs have auth codes on them. For example in the logs here:

Jul 19 06:55:28 Runner(Flutter)[5538] <Notice>: flutter: The Dart VM service is listening on http://127.0.0.1:49886/actyFuKfSr8=/

zanderso avatar Jul 20 '22 04:07 zanderso

cc @bkonyi @a-siva @jason-simmons

zanderso avatar Jul 20 '22 04:07 zanderso

Strange there are a number of other log entries in the output that have prints of VM service URIs that do not have auth codes in them. Is it possible that this particular test is not passing in the disable-service-auth-codes option ?

a-siva avatar Jul 25 '22 15:07 a-siva

That is because the test was flaky.

chinmaygarde avatar Jul 25 '22 17:07 chinmaygarde

That is because the test was flaky

Thanks, yes I did not realize the test was flaky

a-siva avatar Jul 25 '22 17:07 a-siva

From Triage: @a-siva was looking into this but having trouble reproducing it.

chinmaygarde avatar Aug 01 '22 17:08 chinmaygarde

Recent flake stats Mac_ios microbenchmarks_ios flakes five times in recent 100 commits: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5664/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5718/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5727/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5734/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5749/overview

keyonghan avatar Aug 02 '22 16:08 keyonghan

Unable to reproduce this in the stand alone VM, the log seems to indicate this is probably happening when running the engine. Need to check the engine code if there could be some race situation that causes the flag to not be set.

a-siva avatar Aug 08 '22 15:08 a-siva

The flag is passed to Dart_SetVMFlags here: https://github.com/flutter/engine/blob/main/runtime/dart_vm.cc#L430.

The flag is also directly set in the service isolate here: https://github.com/flutter/engine/blob/main/runtime/dart_service_isolate.cc#L197.

Not sure if both of those are necessary, or which one is misbehaving such that it would cause this issue.

zanderso avatar Aug 08 '22 17:08 zanderso

@a-siva Any updates to Zach's query?

chinmaygarde avatar Aug 15 '22 17:08 chinmaygarde

The flag is passed to Dart_SetVMFlags here: https://github.com/flutter/engine/blob/main/runtime/dart_vm.cc#L430.

The flag is also directly set in the service isolate here: https://github.com/flutter/engine/blob/main/runtime/dart_service_isolate.cc#L197.

Not sure if both of those are necessary, or which one is misbehaving such that it would cause this issue.

The first one is not necessary, setting the field _authCodesDisable is the one that takes effect.

a-siva avatar Aug 16 '22 23:08 a-siva

Dans Observation: There is a cocoon agent running on the host which is a Dart process and there is a Dart process running on the device. So one may be using auth codes and the other not. So maybe that is why we are getting confused?

More Triage Observations: The SIGSTOP with all threads idling indicates that the backtrace might be from some port mortem backtrace gathering utility instead of a crash.

chinmaygarde avatar Aug 22 '22 17:08 chinmaygarde

@dnfield I'm not sure that's related. I would suggest that we investigate whether all the flags are passed properly in every possible way that the service isolate might be spawned. If this only repro's on the bots, then we'll have to add more logging, so that we can figure out why https://github.com/flutter/engine/blob/main/runtime/dart_service_isolate.cc#L197 isn't happening.

zanderso avatar Aug 31 '22 00:08 zanderso

Maybe it could also happen that the service isolate could start before the Engine has had a chance to set the flags? Maybe @a-siva can answer whether that's something that could happen. Since this issue is flaky, we should probably be looking for a race of some sort.

zanderso avatar Aug 31 '22 00:08 zanderso

Maybe it could also happen that the service isolate could start before the Engine has had a chance to set the flags? Maybe @a-siva can answer whether that's something that could happen. Since this issue is flaky, we should probably be looking for a race of some sort.

The service isolate start includes a callback into the engine for isolate setup and this is where the engine sets the flag, see https://github.com/flutter/engine/blob/main/runtime/dart_isolate.cc#L753 and https://github.com/flutter/engine/blob/main/runtime/dart_service_isolate.cc#L129 were the field is setup.

a-siva avatar Aug 31 '22 17:08 a-siva

This is the leading cause of flakes in the iOS devicelabs. We are trying to figure out an owner to reason about finding the cause of the flake. I'm going to try to find an owner for this. This is super hard to reproduce and happens pretty infrequently even in the device lab. So this probably involves serious code staring.

chinmaygarde avatar Sep 06 '22 18:09 chinmaygarde

I've tried a local run with https://github.com/flutter/flutter/blob/c5890f0c23eef4fa38e2a574a5d04aa625d2b91b/packages/flutter_tools/lib/src/ios/devices.dart#L354 commented out and the tests still pass.

Something is strnage here. The test harness should still be able to connect regardless of whether the auth code is used or not.

dnfield avatar Sep 07 '22 18:09 dnfield