Specflow+Livingdoc is not generating test results if tests are running in 10 agents in azure release pipeline.

[yogeshsunder]

SpecFlow Version

3.9.8

Which test runner are you using?

MSTest

Test Runner Version Number

2.2.5

.NET Implementation

.NET 5.0

Project Format of the SpecFlow project

Classic project format using packages.config

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

{ "stepAssemblies": [ { "assembly": "XXXXXXXXXXXXX(projectname)" } ], "trace": { "traceSuccessfulSteps": true, "traceTimings": false, "minTracedDuration": "0:0:0.1" }, "livingDocGenerator": { "enabled": true, "filePath": "TestExecution.json" } }

Issue Description

I am trying to get the specflow+living doc for my project in azure release pipeline with the test results. I am using 10 agents in the release pipeline for my tests and specflow+living doc is generating the documentation correct but it is not generating test execution results correctly as it is generating test results only for 1 agent not for all agents. For example I am running 91 test cases in 10 agents and it is giving me test execution result for only last agent which is running only 1 test case. I am using TestExecution*.json for all the test execution files.

Steps to Reproduce

I am trying to get the specflow+living doc for my project in azure release pipeline with the test results. I am using 10 agents in the release pipeline for my tests and specflow+living doc is generating the documentation correct but it is not generating test execution results correctly as it is generating test results only for 1 agent not for all agents. For example I am running 91 test cases in 10 agents and it is giving me test execution result for only last agent which is running only 1 test case. I am using TestExecution*.json for all the test execution files.

Link to Repro Project

No response

[SabotageAndi]

You need to collect all the TestExecution.jsons on one agent and then call the livingdoc cli/task.

[yogeshsunder]

Thanks for your reply Andi, I appreciate it. Will you please elaborate that how I can collect all the TestExecution.jsons on one agent.

[SabotageAndi]

@yogeshsunder Yeah, that depends on your setup. Sadly release pipelines don't have artifacts by default. :-/ There are a lot of tasks in the Azure DevOps Marketplace that lets you upload and download files to the different cloud storage providers. I would use on of those.

[yogeshsunder]

Can you please send me some examples/samples where you have or somebody from your team has done something like that for the reference.

[yogeshsunder]

Hi @SabotageAndi As per your suggestion, I have tried using Azure Blob but there is nothing in the documentations regarding making test execution files on a common storage on azure blob.

Will you please help me understand about the same and let me know where should I write my code for the same whether it is in Diagnostic hooks or anywhere else and how that code will look like.

Also, it there is something else I can use please suggest and a short idea/description regarding the same would be highly appreciated!

Thanks in advance!