Living Doc not getting updated in Azure DevOps even though specflow+LivingDoc task is successful

[avishekjana-89]

SpecFlow Version

3.9.74

Which test runner are you using?

NUnit

Test Runner Version Number

3.13.3

.NET Implementation

.NET Core 3.1

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK

SpecFlow Section in app.config or content of specflow.json

No response

Issue Description

Azure Task Name: SpecFlow + LivingDoc

We are running our specflow projects over azure devops release pipeline, projects are running properly and then we have a task 'SpecFlow+LivingDoc' where it should publish test results from TestExecution.json into living documentation tab of azure. We are getting this steps as passed also. But its not uploading results into azure. Can someone please let me know what could be reasons not to show results in living doc even if uploading is successful showing from azure task.

Interesting point, we are using same steps into another one project. Its uploading results as expected in azure living doc tab.

Steps to Reproduce

1. Create azure release pipeline
2. Create a VSTest task over there
3. Create SpecFlow+LivingDoc task with generated TestExecution.json path

Link to Repro Project

No response

[SabotageAndi]

Could you post the output of the task here?

[avishekjana-89]

Please see output for the specflow build setp task in this.

[SabotageAndi]

Only to be sure. You have selected the correct branch in the branch chooser.

[avishekjana-89]

Yes, definitely, otherwise the code was not pulled from the repo and the previous test task should not be run if the correct branch was not chosen.

[avishekjana-89]

Another point to be added in this ticket, before last Friday this task was working as expected. After that suddenly we see this strange problem. Please be noted that we didn't change version of any code in this meantime.

[SabotageAndi]

Hmm, so the upload works. Could you post two more screenshots?

1. The configuration of the build step
2. this part of your living doc:

[avishekjana-89]

Please find all details as requested. We run tests from the regression branch of the repo, but no living document is published under regression in Azure Devops. Although another pipeline is running with the master branch of the same repo, it is published correctly in ADO.

[SabotageAndi]

Does your release pipeline have a build artifact? There is an issue in Azure DevOps that a release pipeline only gets the branch information when it is connected to a build pipeline artifact as the first dependency, but it doesn't know the branch if it is connected e.g. to a nuget package

[avishekjana-89]

@SabotageAndi Thanks for your response. We are using repository as source under artifact in release pipeline. Even we were able to see living doc for same pipeline till first week of Oct Month also. Interestingly, I have followed same setup steps in my personal azure devops, its working as expected over there.

We investigated issue deeply from our end and observed that all documents are properly stored in local document database. But branches are not showing in dropdown on Living Doc. Even if we try to hit with direct GUID URL on browser, it redirects to a particular living doc report always.

Azure GIT Repo:

Documents API Response: .../_apis/ExtensionManagement/InstalledExtensions/techtalk/techtalk-specflow-plus/Data/Scopes/Default/Current/Collections/Data/Documents

Please see in response that total 23 reports are there for single repo only in local document database.

Document API Response for a particular GUID: .../_apis/ExtensionManagement/InstalledExtensions/techtalk/techtalk-specflow-plus/Data/Scopes/Default/Current/Collections/Data/Documents/{guid}

GUID: ending with .. 7afaa500f1559b95

Living Doc Page:

Only one branch(master) is displaying in branch dropdown of Living doc. GUID is ending with .. 8270112f239b09e for this master branch. Even if we try to hit URL directly on browser for Regression branch with aforementioned GUID(ending with .. 7afaa500f1559b95), it redirects to master branch(GUID ending with ..8270112f239b09e) report only.

[SabotageAndi]

Ok, so your source type of artifact is something else than "Build". That is the reason why it is not working. You need a build as the first artifact in the pipeline. Without that SpecFlow+LivingDoc does not get the branch information it needs.

Docs are here: https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Guides/azdoreleasepipeline.html

[avishekjana-89]

@SabotageAndi Thanks for your response. But it had been working in past when source type of artifact was repository. Even I tried to test with same in my personal azure devops also where source type of artifact is repository, it's working as expected over there, we could see living doc for same.

Can you please help us to understand why now its not working for repository source type then ?

Please see below snapshot from one of my release pipeline in personal azure devOps.

[SabotageAndi]

I have no idea why it worked. It shouldn't have. Can you try it with a build artifact, please? That is also the only officially supported way how to use the LivingDoc task in release pipelines of Azure DevOps.

[avishekjana-89]

@SabotageAndi it worked with build artifact. The issue is being observed when release pipeline is created with repository source type. Can we have application team engaged why it doesn't working?

[SabotageAndi]

The reason is that Azure DevOps doesn't provide the branch information we need if the artifact source is something else than a build artifact. This is a behavior of Azure DevOps that we can't change. Nothing we can do here. Sorry.