very_good_cli icon indicating copy to clipboard operation
very_good_cli copied to clipboard

feat: enable optimization for platform tests

Open ryzizub opened this issue 3 months ago • 3 comments

Description

Currently, the platform tests are running without optimization, which can make the test suite less performant. This is due to inconsistent behavior of platform tests when placed in the same file → https://github.com/VeryGoodOpenSource/very_good_cli/pull/1359/files#r2354792138

To enable optimization, further investigation of flutter test is needed to identify the issue that prevents platform tests from running in the same file.

Requirements

  • [ ] All CI/CD checks are passing.
  • [ ] There is no drop in the test coverage percentage.

Additional Context

No response

ryzizub avatar Sep 17 '25 10:09 ryzizub

During research into what needs to be done to enable this, it seems to come down to two main points:

  1. Updating the test optimizer brick – The test optimizer brick currently contains functionality that is not supported across all platforms. Since this brick would be used on every run, it needs to be updated to either use conditional imports or other methods to support all possible platforms. From the initial tests, it appears the only issue is with the GoldenLocalComparator and its basedir parameter, which is not supported when running on the web.
  2. Test file naming issue – The tests are run from a single test optimizer file named .test_optimizer.dart. For some reason, Flutter does not support test files on the web that begin with a dot. This causes the tests to hang. Once the leading dot is removed, the tests run without issue.

ryzizub avatar Sep 18 '25 07:09 ryzizub

A couple of notes on this issue and the platform parameter.

  • The very good cli already supports passing Flutter-specific parameters, so we should probably remove some of the changes made in #1359.
  • As @ryzizub has already tested, optimization does not work when the platform is specified.
  • The testOn parameter of test function will be supported once we resolve the issues with the optimizer, but the @TestOn(https://api.flutter.dev/flutter/package-test_api_scaffolding/TestOn-class.html) annotation won't be supported by the very_good_cli because annotations are not taken into account in the optimizer file, so we should consider addressing this issue as well (or document it at least).
  • Renaming .test_optimizer.dart to test_optimizer.g.dart should do the trick to fix the issue on web tests.

matiasleyba avatar Sep 26 '25 19:09 matiasleyba

Thanks @matiasleyba !

The very good cli already supports passing Flutter-specific parameters, so we should probably remove some of the changes made in https://github.com/VeryGoodOpenSource/very_good_cli/pull/1359.

Honestly totally forgot about this, but still i see value having it as own parameter since we might do already extra things on top of it and the filtering and working with those i feel is better when we have it seperate. One of the examples is disabling parallel running, that one is disabled when platform param is provided. WDYT? Im open to both ways😄

The testOn parameter of test function will be supported once we resolve the issues with the optimizer, but the @TestOn(https://api.flutter.dev/flutter/package-test_api_scaffolding/TestOn-class.html) annotation won't be supported by the very_good_cli because annotations are not taken into account in the optimizer file, so we should consider addressing this issue as well (or document it at least).

That one will be tricky to support. We might scan for TestOn and group to optimizer that will have that annotation in optimizer. I think documenting it and adding support later is fine for me

Renaming .test_optimizer.dart to test_optimizer.g.dart should do the trick to fix the issue on web tests.

Yeah! I think its way to go as the first step on certain platforms 💪 But its not enough, because there are platform specific things in optimizer that will break on certain platforms we should also fix those (GoldenOptimizer dir param on web)

ryzizub avatar Sep 29 '25 07:09 ryzizub