Plan to add support for integration_test package

[elmickey]

trafficstars

I don't know if the question has already been answered but is there any plan to support integration_test. For me it would help to write steps and help the execution time. The last time, i tried to use gherkin with integration_test was the fact that gherkin needs to read the features files on disk which is incompatible with the way integration_test is working.

[jonsamwell]

I don't have any experience with that package, what would be the benefit?

[elmickey]

I may be wrong but it's the new recommended way to write integration tests https://flutter.dev/docs/testing/integration-tests . But for me the benefits are :

• same syntax as widget test
• can have access to the internal state of the app (would help to write tests which are right now impossible to write with fluttter driver)
• faster tests
• tests can run on firebase test labs

[vrnvorona]

Second this, currently researching what that package does, but it may be solution for a lot of driver problems with same syntax as WT.

[jonsamwell]

@vrnvorona please share your findings. I will have to look into it as well but if it is the way the Flutter team suggest doing integration tests they will probably eventually deprecate the driver. I won't have any time to look at this before the new year unfortunately. So if anyone want to do a proof of concept that would be awesome

[vrnvorona]

@jonsamwell We now need target and driver files, target has main() function with groups and tests. In each test we need to call main() of our app without enabled extensions and not as pumpWidget(app) (otherwise all exceptions of app will break tests). All waits have to be done through pump/pumpAndSettle. All actions are as in WidgetTests.

target

import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:your_app/main.dart' as app;

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();
  group('name', () {
    testWidgets('name', (WidgetTester tester) async {
      app.main();
      await tester.pumpAndSettle();
    });
  } 

driver

import 'package:integration_test/integration_test_driver.dart';

Future<void> main() => integrationDriver();

I can't really tell more on top of this, i tried poking source code of your package, but I lack skills to do it properly as well as understanding in how it all worked before/how it supposed to work now.

I also kinda wonder, what are your plans for after new year? And if you plan to support new package, do you have any estimates for how long before it may be done?

[jonsamwell]

@vrnvorona I have to investigate this to see if it is possible as I think the feature files will need to be assets on the device as at the moment this library is a parent process that invokes the flutter driver process so it is able to read the feature files off disk. The actual swapping of the basic driver commands such as driver.waitFor -> tester.pumpAndSettle() isn't too bad, I will just put a facade over the mechanism to instrument the app and then either the driver or this new integration-tests package can be used.

If this is the way the future of e2e tests are envisaged in flutter then this lib will support it (if possible).

I would be unable to give a timescale for this unfortunately, I have a full time job, and other interests so it is a case of getting some free time but I will try.

[jonsamwell]

I've had a quick look at this and it seems it may be quite hard as the test file is basically launched as the app (as the test file then invokes the app's main method). Therefore, there would be no way for the gherkin lib to read the feature files off the disk. My first though would be to do some sort of code generation which turns the feature files into a single file that can be used for testing. The other method would be to make the feature files assets of the app but that seems wrong. I need to play around with these ideas more.

The actual conversion from the driver api to the tester api won't be hard so I'm confident that is doable.

@elmickey @vrnvorona any thoughts on this?

[jonsamwell]

Hi All,

I have done a very rough first pass at using integration_test. It would be great to get some feedback. See this branch https://github.com/jonsamwell/flutter_gherkin/tree/integration_test__package_support

Look at the changelog for instruction on how to get started and some explanation of the approach.

https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/CHANGELOG.md

Basically as the tests now need to be in the deployed app I have had to use code generation to create the tests at development time so they can be run without need to access the file system etc. There are still some challenges to overcome such as reporting as the build in flutter_test reporter seems absolutely useless. So I am not quite sure how to get the report data from the app to another source. Any idea would be welcome.

Let me know how you get on and any thoughts, suggestions or PR's would be great.

[clpo]

@jonsamwell I've had an opportunity to make use of your integration_test branch. Seems to work great locally but I haven't had a chance to send my app and tests to firebase yet, but can expect it to work. Logging seems fine as well. Nice work! I'll be working more with this next week and i'll let you know if i experience any issues.

[jonsamwell]

@clpo many thanks for the feedback. I'd be interested to see what happens when testing in firebase. I need to get this setup too. I think the biggest problem when testing on firebase will be the reports. I am not sure how to save reports outside of the standard fluttter_test reporting.

On that point I have made the JsonReporter work today when running from flutter drive so update and have a go. You will need to regenerate your tests as well if you do this.

On another note, I suspect there will be some tweaks needed when we start using this in in replace of flutter driver so please add and and all issues you come across.

[clpo]

@jonsamwell i've had chance to run the tests through firebase test labs and it seems to be having issues initiating the tests. The app launches but there is no mention of the driver starting and no interaction is made with the app. I'm not certain if this is a configuration mistake on my part, but this issue does not occur locally. I looked through the logs provided by test labs but i cant pick any particular thing out which would make sense as the cause of the issue. This is just an FYI really.

[vrnvorona]

@jonsamwell regarding reporting, what if we just generate additional steps for reporting purposes? Because approach sounds really different from what was with flutter_driver, I had this blunt stupid idea :)

I will refine my comment when I try your work a little bit

[jonsamwell]

@clpo thanks for trying with firebase. At that stage it is just a normal integration_test app but something might be happening with the way the reporters are doing stuff. I'll try and get a simple test app running in firebase to see what is going on.

@vrnvorona yes, I originally had this idea too so each step would be a test but you cannot have any test calls within a testWidget block unfortunately. I think the reporting is fine locally or on a build server that uses the flutter drive command. I think the problem comes when running against device farm type things like firebase.

[vrnvorona]

@jonsamwell If i understand correctly, we are still using driver exposed methods instead of WidgetTester? I have yet to successfully migrate my integration_test tests onto new flutter_gherkin since i have no idea how to use pump in it etc. I have quite a while of endless animations i need to skip, and waitForAppToSettle() doesn't work same way pumpAndSettle() does. Plus code is not reusable with WidgetTests because we use Driver again 🤔

[jonsamwell]

No it is all using WidgetTester. I had to do an adapter so that the same api could be used for the flutter driver implementations as the WidgetTetser. Flutter driver is only used to actual start the test app. Once the app is under test the appDriver property on the world object uses WidgetTester.

The adapter api is still under flux so of you need something added to it let me know.

See https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/lib/src/flutter/adapters/widget_tester_app_driver_adapter.dart

On Tue, Jan 12, 2021, 22:02 Vladislav Voronin [email protected] wrote:

@jonsamwell https://github.com/jonsamwell If i understand correctly, we are still using driver exposed methods instead of WidgetTester? I have yet to successfully migrate my integration_test tests onto new flutter_gherkin since i have no idea how to use pump in it etc. I have quite a while of endless animations i need to skip, and waitForAppToSettle() doesn't work same way pumpAndSettle() does. Plus code is not reusable with WidgetTests because we use Driver again 🤔

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jonsamwell/flutter_gherkin/issues/101#issuecomment-758580699, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4F7ILNWKVKLJDL5PCSG3LSZQT5XANCNFSM4UUB6Y4A .

[vrnvorona]

@jonsamwell Is it possible/viable to just expose WidgetTester through world/context? Or it is bad idea? For me the most appealing part of new package aside from access to better api (like keyboard event buttons) is code similarity with WidgetTests themselves, possibly allowing to write tests for E2E and Widgets faster than with two different approaches.

[vrnvorona]

Btw while i was investigating, found little errors in changelog, here is PR https://github.com/jonsamwell/flutter_gherkin/pull/109 (for md file lmao but still) so others who use it won't have to debug.

[jonsamwell]

@vrnvorona many thanks for the PR much appreciated! I'll get that merged in.

The WidgetTester is exposed via the world object, you can use world.appDriver.rawDriver however, this is not typed to WidgetTester so I'll see if I can type it. (see https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/lib/src/flutter/adapters/app_driver_adapter.dart#L21)

Sorry for lack of documentation around this, once we have things working I'll focus on API documentation and updating the readme etc

[vrnvorona]

So i can assume rawDriver as WidgetTester and work with it same way? Thanks, i cam to this conclusion too when checking adapter files, but didn't verify it yet mostly because had issues with app starting.

Many thanks again for quick work on adaptation.

[jonsamwell]

Yes this is safe to assume, I will try and get a World with strongly typed adapter so intellisense will work on it

[vrnvorona]

@jonsamwell I have this step with custom timeout (because 10 secs is not enough), but when i run step, i get null config and it uses default duration.

when1<int, FlutterWorld>('I wait {int} seconds with animation', (duration, context) async {
          final tester = context.world.appDriver.rawDriver as WidgetTester;
          try {
            await tester.pumpAndSettle(
                const Duration(milliseconds: 100), EnginePhase.sendSemanticsUpdate, Duration(seconds: duration));
            // ignore: avoid_catching_errors
          } on FlutterError {
            // pump for 2 seconds and stop
          }
        }, configuration: StepDefinitionConfiguration()..timeout = const Duration(minutes: 5)),

I checked StepDefinitionGeneric constructor and found that when steps are declared, they have config, but if i check run function they don't. In runStep(), because there they also all have null config, but i didn't yet narrow problem any further.

[jonsamwell]

Thanks @vrnvorona I'll take a look at this today

[vrnvorona]

Thanks @vrnvorona I'll take a look at this today

Found that issue was not in that.

I have two declared step defs: "I wait {int} seconds" and "I wait {int} seconds with animation" (one is looping pump, other is puming with set timeout, then catches FlutterError so i can wait some time with infinite animation), and firstMatch method returns "I wait {int} seconds" instead of "I wait {int} seconds with animation" despite my feature file and generated code having runStep("I wait 5 seconds with animation").

Not entirely correct behaviour. Sorry for false lead above.

When i commented "I wait {int} seconds" step def, everything is working fine, even the original issue with timeout was cause by infinite pumpAndSettle() because wrong step was used. I need to be more careful with debugging :(

Update: fixed by using regexp with $ on end. For some reason ^ as for beginning of string doesn't work.

[vrnvorona]

@jonsamwell btw, did you notice that tests now run from root folder or it's something with my setup? I tried to use json reporter and noticed that CWD is always '/' while tests are running. Which ofc breaks './report.json' thing

[jonsamwell]

@vrnvorona Yes I couldn't replicate your issue with the timeout so glad you fixed it. I actually added a test in for that in the example (see https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/integration_test/gherkin/steps/when_await_animation.dart)

I am not sure what you mean about the CWD but any reporter that writes to the file system will not work with this as the tests are actually run on the app and not the host, so the app will not be able to save any files outside of the app file system drive. As a work around the JsonReporter can now be serialise the report and send it over the wire. See the example but you need to make the write operation on the JsonReporter a noop (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/integration_test/gherkin/configuration.dart#L35) then the results will be sent back to the host and saved as here (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/test_driver/integration_test_driver.dart#L22)

Perhaps you are not using the latest commit to this branch? Add this to your pubspec.yaml:

  flutter_gherkin:
    git:
      url: https://github.com/jonsamwell/flutter_gherkin.git
      ref: 230a8e8d5fcc86d8544b9a196f611a375bab4a0c

[jonsamwell]

I have this new branch running for a really complex suite of 100 test in a real life application now and it works really well and it is fast and more stable that the flutter_driver implementation. There is still quite a bit of work to do updating the documentation but I think this is pretty much code complete now.

[jonsamwell]

Also with the latest the appDriver property and rawDriver properties are now correctly typed to either WidgetTester or FlutterDriver. If you use WidgetTester your world needs to extend FlutterWidgetTesterWorld (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/lib/src/flutter/world/flutter_widget_tester_world.dart#L9). On a side note I have been able to do over 100 complex tests without the need to use the WidgetTester directly but relying on the AppDriverAdapter instance on the world context. I think this is best practice for flexibility and code correctness.

Scenarios are also correctly filtered by tags when run now

[vrnvorona]

@jonsamwell Thanks for explanation on how reports work, I just couldn't find that report because I totally forgot about that path and was looking for usual report.json in root of project. Now I can find report.

As for rawDriver I will check latest commit, but so far I just did final tester = context.world.appDriver.rawDriver as WidgetTester and it worked.

So far we don't have a lot of tests since we prototype architecture for how we will use them for apps and so far we think that having same syntax as literal WidgetTests can be beneficial (allows us to have all finders in one folder for elements we need etc), but we would adapt if needed.

I am glad to hear that tests work good and better than flutter_driver, very good news.

[vrnvorona]

As for WidgetTester again, i just change when<FlutterWorld> with when<FlutterWidgetTesterWorld> and it's good to go?

[jonsamwell]

Yes that is correct

On Fri, Jan 15, 2021, 21:44 Vladislav Voronin [email protected] wrote:

As for WidgetTester again, i just change when<FlutterWorld> with when<FlutterWidgetTesterWorld> and it's good to go?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jonsamwell/flutter_gherkin/issues/101#issuecomment-760834457, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4F7IPRBHOOVA4EGWHXITTS2AL7RANCNFSM4UUB6Y4A .