Helix failing during azure_devops_result_publisher

Open ericstj opened this issue 1 year ago • 0 comments

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=643511

Build leg reported

Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution

Pull Request

https://github.com/dotnet/runtime.git/pull/101047

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "azure.devops.exceptions.AzureDevOpsServiceError: TF10216: Azure DevOps services are currently unavailable. Try again later. ",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

[ ] Feature changes/additions
[ ] Bug fixes
[ ] Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-101047-merge-a1e320a218ce40c5a9/Microsoft.Bcl.AsyncInterfaces.Tests.Attempt.3/10721bf8-d0ef-4acb-921e-16e8c91f37f7.log?helixlogtype=result

2024-04-15T12:13:15.744Z	INFO   	interval(67)	set_abort_after_timer	Set the timer's abort_after to 1260 seconds from now: abort_after = 2024-04-15 12:34:15.744366
2024-04-15T12:13:39.424Z	ERROR  	azure_devops_result_publisher(164)	log_error	got error: Traceback (most recent call last):
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 169, in upload
    self._process(batch)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 181, in _process
    hot_path_tests += self._publish_results(converted)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 330, in _publish_results
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 295, in _send
    raise ex
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 271, in _send
    return execute()
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 330, in <lambda>
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\azure\devops\v5_1\test\test_client.py", line 675, in add_test_results_to_test_run
    response = self._send(http_method='POST',
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 70, in _hook_send
    return old_send(*args, **kwargs)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 104, in _send
    response = self._send_request(request=request, headers=headers, content=content, media_type=media_type)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 68, in _send_request
    self._handle_error(request, response)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 247, in _handle_error
    raise AzureDevOpsServiceError(wrapped_exception)
azure.devops.exceptions.AzureDevOpsServiceError: TF10216: Azure DevOps services are currently unavailable. Try again later. Activity Id: 9c3f2a7f-c6b1-44dd-a824-dca23cc5b579
Traceback (most recent call last):
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 169, in upload
    self._process(batch)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 181, in _process
    hot_path_tests += self._publish_results(converted)
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 330, in _publish_results
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 295, in _send
    raise ex
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 271, in _send
    return execute()
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 330, in <lambda>
    published_results = self._send(lambda: test_client.add_test_results_to_test_run(
  File "C:\h\scripts\helix-scripts\azure\devops\v5_1\test\test_client.py", line 675, in add_test_results_to_test_run
    response = self._send(http_method='POST',
  File "C:\h\scripts\helix-scripts\helix\test_reporting\azure_devops_result_publisher.py", line 70, in _hook_send
    return old_send(*args, **kwargs)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 104, in _send
    response = self._send_request(request=request, headers=headers, content=content, media_type=media_type)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 68, in _send_request
    self._handle_error(request, response)
  File "C:\h\scripts\helix-scripts\azure\devops\client.py", line 247, in _handle_error
    raise AzureDevOpsServiceError(wrapped_exception)
azure.devops.exceptions.AzureDevOpsServiceError: TF10216: Azure DevOps services are currently unavailable. Try again later. Activity Id: 9c3f2a7f-c6b1-44dd-a824-dca23cc5b579
2024-04-15T12:13:39.439Z	INFO   	event(51)	send	Sending event type Error
2024-04-15T12:13:39.564Z	INFO   	saferequests(87)	request_with_retry	Response complete with status code '201'
2024-04-15T12:13:39.564Z	INFO   	logs(165)	__exit__	Uploading self-uploading log file C:\h\logs\10721bf8-d0ef-4acb-921e-16e8c91f37f7.log (Client: a000ZGO)

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=643511 Error message validated: [azure.devops.exceptions.AzureDevOpsServiceError: TF10216: Azure DevOps services are currently unavailable. Try again later. ] Result validation: :x: Known issue did not match with the provided build. Validation performed at: 4/15/2024 7:23:15 PM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Apr 15 '24 19:04 ericstj