[golden_toolkit] Goldens for different devices have different time advance

[cyrax111]

I made a simple project just entered : flutter create golden

and added FutureBuilder

FutureBuilder(
  future: Future<int>.delayed(
      const Duration(milliseconds: 150), () => 1),
  builder: (context, snapshot) {
    if (snapshot.hasData) {
      return Container(
          color: Colors.orange, width: 100, height: 100);
    }
    return const CircularProgressIndicator();
  },
)

The behavior is the first 150 milliseconds is showed CircularProgressIndicator then an orange container.

I wrote follow golden test:

testGoldens('Counter golden test', (WidgetTester tester) async {
  await loadAppFonts();

  await tester.pumpWidgetBuilder(const MyHomePage(title: 'Title'));

  await multiScreenGolden(
    tester,
    'my_home_page',
    devices: [Device.phone, Device.iphone11, Device.tabletLandscape],
    customPump: (tester) async {},
    deviceSetup: (device, tester) async {
      await tester.pump(const Duration(milliseconds: 50));
    },
  );

  await tester.pump(const Duration(hours: 1));
});

and have got follow result: Device.phone

Device.iphone11

Device.tabletLandscape

Despite only one tester.pump I have different time advances.

I expected the same picture on all devices like on Device.phone.

[coreysprague]

the device setup function runs in between resizing for each device. So in this case, 50ms are pumped before Device.phone, 100ms are pumped before iphone11, and 150ms are pumped before the tablet landscape golden. In other words, what you are seeing is working as intended.

if your deviceSetup function does not advance time, I would expect all of the screen variations to be in the same state.

[cyrax111]

if I leave deviceSetup empty and put the pump in customPump the result will be the same:

  await multiScreenGolden(
    tester,
    'my_home_page',
    devices: [Device.phone, Device.iphone11, Device.tabletLandscape],
    customPump: (tester) async {
      await tester.pump(const Duration(milliseconds: 50));
    },
    deviceSetup: (device, tester) async {
    },
  );

[coreysprague]

I think the customPump (I HATE that name) will similarly be called in between each golden as well.

What are you trying to do, are you trying to pump 50ms and THEN capture a golden for multiple devices? If so, just move your await tester.pump(const Duration(milliseconds: 50)); to be before you call await multiScreenGolden

[coreysprague]

or you could do something like:

await tester.pump(const Duration(milliseconds: 50));
await multiScreenGolden(tester, 'loading' ... )


await tester.pump(const Duration(milliseconds: 100));
await multiScreenGolden(tester, 'loaded' ... );

[cyrax111]

I need to rid from any pumps between each golden. If I leave deviceSetup and customPump empty and move my pump before multiScreenGolden I have got an error.

[cyrax111]

the error:

00:04 +0: Counter golden test                                                                                           
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown running a test:
'package:flutter_test/src/_matchers_io.dart': Failed assertion: line 28 pos 10:
'!renderObject.debugNeedsPaint': is not true.

When the exception was thrown, this was the stack:
#2      captureImage (package:flutter_test/src/_matchers_io.dart:28:10)
#3      MatchesGoldenFile.matchAsync (package:flutter_test/src/_matchers_io.dart:82:21)
#4      _expect (package:test_api/src/expect/expect.dart:101:26)
#5      expectLater (package:test_api/src/expect/expect.dart:62:5)
#6      expectLater (package:flutter_test/src/widget_tester.dart:492:23)
#7      compareWithGolden (package:golden_toolkit/src/testing_tools.dart:290:9)
<asynchronous suspension>
<asynchronous suspension>
(elided 3 frames from class _AssertionError and package:stack_trace)

The test description was:
  Counter golden test
════════════════════════════════════════════════════════════════════════════════════════════════════
00:04 +0 -1: Counter golden test [E]                                                                                    
  Test failed. See exception logs above.
  The test description was: Counter golden test

For easy access I made a repository https://github.com/cyrax111/golden_test

[coreysprague]

I'll try to look at it today. I suspect you will still need a pump, but with Duration(milliseconds: 0)

[cyrax111]

Thank you! I think one pump without duration don't help me unfortunately as well. You can check the tag v2 for it.

[cyrax111]

@coreysprague you can try this https://github.com/cyrax111/golden_test/tree/v2

[coreysprague]

What if you do this?

Obviously the counts are wrong -- in order to multiScreenGoldens to be captured, the widgets have to be rebuilt, which in your code will always increment the counter...

But:

[coreysprague]

@cyrax111 -- sorry for the delay. Been slammed at work

[cyrax111]

@coreysprague thank you for your answer!!

[cyrax111]

in order to multiScreenGoldens to be captured, the widgets have to be rebuilt

I think I need a little bit another behavior. I want to separate devices in order to make them independent.

I added this feature here https://github.com/cyrax111/golden_test/tree/v3. There is testDeviceGoldens in test_device_goldens.dart file. And you can run widget_separate_devices_test.dart for test.

If you want I can make PR.

Result: