azure-flutter-tasks
azure-flutter-tasks copied to clipboard
hey24sheep
Flutter test - Test report is showing wrong count of test cases.
I am running Flutter test command with publish report and code coverage report enabled. While running the task I get same result as in visual studio terminal (EG: 96 passed 1 failed)
but in reports on azure pipeline it is showing me 134 test cases and all passed.
Attaching snapshots.]
Kindly help me with issue. if anything else is needed please let me know.
Hi, I apologize for the delay. "Test" task was rewritten (as there were same issues with it that you have just reported) and was tested by the community (refer to this issue). No one has reported any issue with it.
I can't help much as I don't know where it could be failing and how can it be failing. As small unit tests to test the "test task" are all passing and community also said it's working fine. I will keep this issue opened up if in case anyone else maybe help here
<testsuites>
<testsuite timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="1">
<testcase name="Some tests failed." classname="Some tests failed." time="0"/>
</testsuite>
<testsuite name="auth_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="7">
<testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/auth_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/auth_model_test.dart" time="9.235"/>
<testcase name="AuthModelNotifier Tests Set AuthModel" classname="AuthModelNotifier Tests Set AuthModel" time="0.032"/>
<testcase name="AuthModelNotifier Tests Clear AuthModel" classname="AuthModelNotifier Tests Clear AuthModel" time="0"/>
<testcase name="AuthModel Tests AuthModel toMap" classname="AuthModel Tests AuthModel toMap" time="0"/>
<testcase name="AuthModel Tests AuthModel fromMap" classname="AuthModel Tests AuthModel fromMap" time="0.015"/>
<testcase name="AuthModel Tests AuthModel toJson" classname="AuthModel Tests AuthModel toJson" time="0"/>
<testcase name="AuthModel Tests AuthModel fromJson" classname="AuthModel Tests AuthModel fromJson" time="0"/>
</testsuite>
<testsuite name="device_info_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="2">
<testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/device_info_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/device_info_model_test.dart" time="0.759"/>
<testcase name="DeviceInfoModel fromMap should create a DeviceInfoModel from a map" classname="DeviceInfoModel fromMap should create a DeviceInfoModel from a map" time="0"/>
</testsuite>
<testsuite name="organization_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="4">
<testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/organization_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/organization_model_test.dart" time="0.764"/>
<testcase name="OrganisationModel toJson and fromJson" classname="OrganisationModel toJson and fromJson" time="0.015"/>
<testcase name="OrganisationModel toMap and fromMap" classname="OrganisationModel toMap and fromMap" time="0"/>
<testcase name="OrganisationModel toString" classname="OrganisationModel toString" time="0"/>
</testsuite>
<testsuite name="role_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="8">
<testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/role_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/role_model_test.dart" time="0.908"/>
<testcase name="RoleModel Tests RoleModel should be created with default values" classname="RoleModel Tests RoleModel should be created with default values" time="0.031"/>
<testcase name="RoleModel Tests RoleModel should be created with values" classname="RoleModel Tests RoleModel should be created with values" time="0"/>
<testcase name="RoleModel Tests RoleModel should be created from a map" classname="RoleModel Tests RoleModel should be created from a map" time="0"/>
<testcase name="RoleModel Tests RoleModel should be converted to a map" classname="RoleModel Tests RoleModel should be converted to a map" time="0"/>
<testcase name="RoleModel Tests RoleModel should be created from JSON" classname="RoleModel Tests RoleModel should be created from JSON" time="0.015"/>
<testcase name="RoleModel Tests RoleModel should be converted to JSON" classname="RoleModel Tests RoleModel should be converted to JSON" time="0"/>
<testcase name="RoleModel Tests RoleModel toString() method should return a formatted string" classname="RoleModel Tests RoleModel toString() method should return a formatted string" time="0"/>
</testsuite>
<testsuite name="session_data_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="5">
<testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_data_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_data_model_test.dart" time="0.734"/>
<testcase name="SessionDataModel tests Creating SessionDataModel instance" classname="SessionDataModel tests Creating SessionDataModel instance" time="0.03"/>
<testcase name="SessionDataModel tests CopyWith method" classname="SessionDataModel tests CopyWith method" time="0"/>
<testcase name="SessionDataModel tests ToMap and fromMap methods" classname="SessionDataModel tests ToMap and fromMap methods" time="0"/>
<testcase name="SessionDataModel tests ToJson and fromJson methods" classname="SessionDataModel tests ToJson and fromJson methods" time="0"/>
</testsuite>
<testsuite name="session_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="3">
<testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_model_test.dart" time="0.711"/>
<testcase name="SessionModel Can create SessionModel instance" classname="SessionModel Can create SessionModel instance" time="0.03"/>
<testcase name="SessionModel Can set and get properties" classname="SessionModel Can set and get properties" time="0"/>
</testsuite>
<testsuite name="auth_storage_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="5">
<testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/providers/auth_storage_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/providers/auth_storage_test.dart" time="0.857"/>
<testcase name="AuthStorage Tests Save and load token from cache" classname="AuthStorage Tests Save and load token from cache" time="0.03"/>
<testcase name="AuthStorage Tests Load isLoggedIn from cache" classname="AuthStorage Tests Load isLoggedIn from cache" time="0"/>
<testcase name="AuthStorage Tests Save and load device token from cache" classname="AuthStorage Tests Save and load device token from cache" time="0.015"/>
<testcase name="AuthStorage Tests Clear storage" classname="AuthStorage Tests Clear storage" time="0"/>
</testsuite>
Abve is my junit.xml file ... If I read this file I get extra test case like "loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/providers/auth_storage_test.dart for every file "_test.dart" file , Also there is a test case for - <testcase name="Some tests failed." I am using version 0 of Flutter Task extension. Is there any other version to update?
"Some tests failed" is generated, that we can't remove because internally it works by reading the STDOUT of the terminal and makes test cases out of that. Looking at the file, I see that the file is generated correctly, each test is having 1 extra case which is the "loading" one, that is what you are saying? But, I think that is correct way of the file, no?
I will keep it open if anyone from the community can help here.
Version for "Flutter Test Task" should be "0.3.5"
We are seeing the same as the OP in our pipeline. The published test report, that shows in the Test tab is indicating that all tests have passed but the reality is that some tests are failing.
@mikoedg Hi, I can understand and I appologize. The problem is with the parser and how Azure gives the STDOUT data. I can try and test but I don't have a project in Flutter anymore. I will keep this open if anyone from the community can help.
@tylerbarnesTrexis No update on this, I can't figure out how to fix it. So if anyone can help me fix it that will glad. I don't know if the success rate is extra or failing rate is extra.
Did some minor digging around for this out of curiosity. If tasks/test/index.js createTestCase() is modified to something like this, I think the results might be accurate based on the test cases in sample project.
function createTestCase(suite, output, globalFailure) {
var testRunRegex = /\s*\d\d:\d\d (\+\d+)?(\s+\-\d+)?:\s*(.*)/;
var match = testRunRegex.exec(output);
if (!match || match.length < 4) {
return suite;
}
var tSplits = match[3].split(': ');
if (tSplits.length === 1) {
// Log printed starting a new test suite - No test case handled here
return suite;
}
// Sanitise test case name from other information
let title = tSplits[tSplits.length - 1];
title = title.replace(' [E]', '');
const currentCase = suite.cases.find(testCase => testCase.title === title);
if (!currentCase) {
// Suite has to be marked as succeeded first, marking as failed can be done when there is a new row with [E] at the end
suite.cases.push({
title: title.trim(),
isSuccess: true,
started: new Date(),
ended: new Date(),
});
suite.succeeded += 1;
return suite;
}
if (match[3].endsWith(' [E]')) {
currentCase.isSuccess = false;
suite.failed += 1;
suite.succeeded -= 1;
}
return suite;
}
It looked to me like there were two test case results for cases which fail. That being said, I don't know anything about the test runner and parsing these results (this was just a small naive test).
Edit: To clarify, each individual case showed as succeeded, each error row of the output was its own test case as well as the "loading.." text. Test cases before the change:
{
"isSuccess": false,
"suites": [
{
"title": "1_api_test.dart",
"isSuccess": false,
"succeeded": 6,
"failed": 0,
"cases": [
{
"title": "loading C:/Git/azure-flutter-tasks/sample_project/test/1_api_test.dart",
"isSuccess": true,
"started": "2024-09-25T10:35:13.566Z",
"ended": "2024-09-25T10:35:16.075Z"
},
{
"title": "1_api_t1",
"isSuccess": true,
"started": "2024-09-25T10:35:16.075Z",
"ended": "2024-09-25T10:35:16.098Z"
},
{
"title": "1_api_t2",
"isSuccess": true,
"started": "2024-09-25T10:35:16.098Z",
"ended": "2024-09-25T10:35:16.102Z"
},
{
"title": "1_api_t3",
"isSuccess": true,
"started": "2024-09-25T10:35:16.102Z",
"ended": "2024-09-25T10:35:16.104Z"
},
{
"title": "1_api_fail_t4",
"isSuccess": true,
"started": "2024-09-25T10:35:16.104Z",
"ended": "2024-09-25T10:35:16.167Z"
},
{
"title": "1_api_fail_t4 [E]",
"isSuccess": true,
"started": "2024-09-25T10:35:16.167Z",
"ended": "2024-09-25T10:35:16.167Z"
}
]
},
{
"title": "2_api_test.dart",
"isSuccess": false,
"succeeded": 1,
"failed": 0,
"cases": [
{
"title": "2_api_t1",
"isSuccess": true,
"started": "2024-09-25T10:35:15.964Z",
"ended": "2024-09-25T10:35:15.964Z"
}
]
},
{
"isSuccess": false,
"succeeded": 1,
"failed": 0,
"cases": [
{
"title": "Some tests failed.",
"isSuccess": true,
"started": "2024-09-25T10:35:17.881Z",
"ended": "2024-09-25T10:35:17.881Z"
}
]
},
{
"title": "3_services_test.dart",
"isSuccess": false,
"succeeded": 2,
"failed": 0,
"cases": [
{
"title": "3_service_t1",
"isSuccess": true,
"started": "2024-09-25T10:35:16.173Z",
"ended": "2024-09-25T10:35:16.212Z"
},
{
"title": "3_service_t2",
"isSuccess": true,
"started": "2024-09-25T10:35:16.212Z",
"ended": "2024-09-25T10:35:16.212Z"
}
]
},
{
"title": "4_widget_test.dart",
"isSuccess": false,
"succeeded": 3,
"failed": 0,
"cases": [
{
"title": "4_widget_t1",
"isSuccess": true,
"started": "2024-09-25T10:35:16.430Z",
"ended": "2024-09-25T10:35:16.981Z"
},
{
"title": "4_widget_t2",
"isSuccess": true,
"started": "2024-09-25T10:35:16.981Z",
"ended": "2024-09-25T10:35:17.030Z"
},
{
"title": "4_widget_t3",
"isSuccess": true,
"started": "2024-09-25T10:35:17.030Z",
"ended": "2024-09-25T10:35:17.030Z"
}
]
},
{
"title": "5_widget_test.dart",
"isSuccess": false,
"succeeded": 1,
"failed": 0,
"cases": [
{
"title": "5_widget_t1",
"isSuccess": true,
"started": "2024-09-25T10:35:17.320Z",
"ended": "2024-09-25T10:35:17.320Z"
}
]
}
]
}
And with the change:
{
"isSuccess": false,
"suites": [
{
"title": "1_api_test.dart",
"isSuccess": false,
"succeeded": 3,
"failed": 1,
"cases": [
{
"title": "1_api_t1",
"isSuccess": true,
"started": "2024-09-25T16:33:03.686Z",
"ended": "2024-09-25T16:33:03.686Z"
},
{
"title": "1_api_t2",
"isSuccess": true,
"started": "2024-09-25T16:33:03.741Z",
"ended": "2024-09-25T16:33:03.741Z"
},
{
"title": "1_api_t3",
"isSuccess": true,
"started": "2024-09-25T16:33:03.748Z",
"ended": "2024-09-25T16:33:03.748Z"
},
{
"title": "1_api_fail_t4",
"isSuccess": false,
"started": "2024-09-25T16:33:03.756Z",
"ended": "2024-09-25T16:33:03.756Z"
}
]
},
{
"title": "2_api_test.dart",
"isSuccess": false,
"succeeded": 1,
"failed": 0,
"cases": [
{
"title": "2_api_t1",
"isSuccess": true,
"started": "2024-09-25T16:33:04.108Z",
"ended": "2024-09-25T16:33:04.108Z"
}
]
},
{
"title": "3_services_test.dart",
"isSuccess": false,
"succeeded": 2,
"failed": 0,
"cases": [
{
"title": "3_service_t1",
"isSuccess": true,
"started": "2024-09-25T16:33:04.487Z",
"ended": "2024-09-25T16:33:04.487Z"
},
{
"title": "3_service_t2",
"isSuccess": true,
"started": "2024-09-25T16:33:04.547Z",
"ended": "2024-09-25T16:33:04.547Z"
}
]
},
{
"title": "4_widget_test.dart",
"isSuccess": false,
"succeeded": 1,
"failed": 0,
"cases": [
{
"title": "4_widget_t1",
"isSuccess": true,
"started": "2024-09-25T16:33:05.014Z",
"ended": "2024-09-25T16:33:05.014Z"
}
]
},
{
"title": "5_widget_test.dart",
"isSuccess": false,
"succeeded": 1,
"failed": 0,
"cases": [
{
"title": "5_widget_t1",
"isSuccess": true,
"started": "2024-09-25T16:33:06.651Z",
"ended": "2024-09-25T16:33:06.651Z"
}
]
}
]
}
Hi all, @ipuhakka thanks for the code I incorporated it and release a version 0.4.4. Please check and let me know if it fixed the issue. @tylerbarnesTrexis @mikoedg @komalkhatri08
Hi, can anyone update me so I can close this issue if it's fixed. Thank you
I am seeing this issue as well using 0.4.4 version of the extension. I only have 5 unit tests right now (new project) but it is showing 25. I notice the task says version 0.4.3 though?
@hey24sheep Can you elaborate on the version of this fix, I take it that using [email protected] should contain the fix? I checked quickly that using task version 0.4.1 publishes the old wrong results, but 0.4.2 does not publish results at all. Possible that this caused some other problems that aren't visible when testing with the sample project locally?
I am seeing this issue as well using 0.4.4 version of the extension. I only have 5 unit tests right now (new project) but it is showing 25. I notice the task says version 0.4.3 though?
Extension version should be 0.4.4 and when you run the task it should say [email protected]
@ipuhakka latest release version is 0.4.2 for flutter test task. Can you pass in publishTests as true explicitly and see if it generates the results
@hey24sheep Just tried it. Still the same problem, 'No Result Found to Publish'.
Is there any update on this? We are using the 0.4.4 and we still having the issue
Hi, no update as I am unable to reproduce and see what is actually happening. I will need some help here, I have already rewritten the whole logic around the test task but it didn't help much. It works locally and around my tests so I am unable to reproduce as to what is actually happening. Community help would be great
Our test results only show a count of successful tests. We added one failure as a test, it it reports 0 failures.
As an alternative you can run the test with powershell and convert the result using something like this https://github.com/TOPdesk/dart-junitreport/issues/45#issuecomment-2911597655 Beware that, currently, this converter creates a file declaring xml version 1.1, which Devops refuses to read. So you have to change to 1.0 in a local copy or a fork.
I opted for a local copy of the project in ./tools/tojunit. And here is my pipeline for tests:
- powershell: |
flutter test --file-reporter=json:test_reports/test.jsonl
cd tools/tojunit
flutter pub get
cd ../..
dart run ./tools/tojunit/bin/tojunit.dart --input test_reports/test.jsonl --output test_reports/TEST-test.xml
displayName: Run tests
- task: PublishTestResults@2
displayName: Publish test results
condition: succeededOrFailed()
inputs:
searchFolder: $(Build.SourcesDirectory)/test_reports
As an added bonus, test suites are properly handled and tests can be grouped by Test file.
