nuget authentication issues from version 1.25

[cyberblast]

Describe the bug

Hi, we are having sudden issues with Dependabot in Azure DevOps since 1.25. We did not change anything on the setup but it is simply failing.

Here's some excerpt from the Pipe log:

[...] 2024-01-05T02:04:40.5358436Z Finding updated dependencies for Microsoft.Extensions.Configuration.Abstractions. 2024-01-05T02:04:40.5371797Z 🌍 --> GET https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json 2024-01-05T02:04:40.6504104Z 🌍 <-- 200 https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json 2024-01-05T02:04:40.7677354Z 🌍 --> GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.configuration.abstractions/8.0.0/microsoft.extensions.configuration.abstractions.nuspec 2024-01-05T02:04:40.9643329Z 🌍 <-- 200 https://api.nuget.org/v3-flatcontainer/microsoft.extensions.configuration.abstractions/8.0.0/microsoft.extensions.configuration.abstractions.nuspec 2024-01-05T02:04:41.0957083Z 🌍 --> GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.primitives/8.0.0/microsoft.extensions.primitives.nuspec 2024-01-05T02:04:41.2007735Z 🌍 <-- 200 https://api.nuget.org/v3-flatcontainer/microsoft.extensions.primitives/8.0.0/microsoft.extensions.primitives.nuspec 2024-01-05T02:04:41.3605608Z Updating Microsoft.Extensions.Configuration.Abstractions from 6.0.0 to 2024-01-05T02:04:41.3621344Z running NuGet updater: 2024-01-05T02:04:41.3622843Z /opt/nuget/NuGetUpdater/NuGetUpdater.Cli update --repo-root /home/dependabot/dependabot-updater/tmp/TITAN-Products-Publicis/Publicis%20OS/_git/PM.CF.DictionarySync.Service --solution-or-project /home/dependabot/dependabot-updater/tmp/TITAN-Products-Publicis/Publicis%20OS/_git/PM.CF.DictionarySync.Service/src/PM.CF.DictionarySync.MdmSync.Service.Application/PM.CF.DictionarySync.MdmSync.Service.Application.csproj --dependency Microsoft.Extensions.Configuration.Abstractions --new-version 8.0.0 --previous-version 6.0.0 --verbose 2024-01-05T02:07:12.0654101Z Updating global.json files. 2024-01-05T02:07:12.0655097Z Dependency [Microsoft.Extensions.Configuration.Abstractions] not found in any global.json files. 2024-01-05T02:07:12.0655904Z No dotnet-tools.json files found. 2024-01-05T02:07:12.0656726Z Running for project [/home/dependabot/dependabot-updater/tmp/TITAN-Products-Publicis/Publicis%20OS/_git/PM.CF.DictionarySync.Service/src/PM.CF.DictionarySync.MdmSync.Service.Application/PM.CF.DictionarySync.MdmSync.Service.Application.csproj] 2024-01-05T02:07:12.0658338Z Running for SDK-style project 2024-01-05T02:07:12.0661015Z dotnet build in GetAllPackageDependenciesAsync failed. STDOUT: MSBuild version 17.8.3+195e7f5a3 for .NET 2024-01-05T02:07:12.0661358Z Determining projects to restore... 2024-01-05T02:07:12.0661928Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0662653Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0663381Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0664101Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0665039Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0665755Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0666480Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0667193Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0667983Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0668691Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0669416Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0670127Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0670823Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0671533Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0672373Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0673233Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0673875Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0674511Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0675219Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0675928Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0676712Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0677429Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0678139Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0678855Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0679552Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0680377Z /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. [/tmp/package-dependency-resolution_t2ntNl/Project.csproj] 2024-01-05T02:07:12.0681140Z /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_t2ntNl/Project.csproj] 2024-01-05T02:07:12.0681414Z 2024-01-05T02:07:12.0681618Z Build FAILED. 2024-01-05T02:07:12.0681691Z 2024-01-05T02:07:12.0682190Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0682818Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0683416Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0684034Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0684826Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0685525Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0686233Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0686932Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0687645Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0688403Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0689186Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0689906Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0690616Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0691320Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0692020Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0692741Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0693451Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0694162Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0694858Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0695561Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0696346Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0697068Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0697763Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0698466Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0699175Z /tmp/package-dependency-resolution_t2ntNl/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. 2024-01-05T02:07:12.0699982Z /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json. [/tmp/package-dependency-resolution_t2ntNl/Project.csproj] 2024-01-05T02:07:12.0700775Z /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_t2ntNl/Project.csproj] 2024-01-05T02:07:12.0701125Z 0 Warning(s) 2024-01-05T02:07:12.0701285Z 26 Error(s) 2024-01-05T02:07:12.0701355Z 2024-01-05T02:07:12.0701536Z Time Elapsed 00:02:29.75 2024-01-05T02:07:12.0701686Z 2024-01-05T02:07:12.0701839Z STDERR: 2024-01-05T02:07:12.0701922Z 2024-01-05T02:07:12.0702581Z Package [Microsoft.Extensions.Configuration.Abstractions] Does not exist as a dependency in [/home/dependabot/dependabot-updater/tmp/TITAN-Products-Publicis/Publicis%20OS/_git/PM.CF.DictionarySync.Service/src/PM.CF.DictionarySync.MdmSync.Service.Application/PM.CF.DictionarySync.MdmSync.Service.Application.csproj]. 2024-01-05T02:07:12.0703027Z Update complete. [...]

I also recognized, that it is suddenly fetching > 1000 nuget dependency files. Where as before it were only like ~90.

Eventually the pipe has been cancelled by DevOps due to long run execution.

##[error]The job running on agent Hosted Agent ran longer than the maximum time of 60 minutes.

I also doublechecked permission of build service on the private Artifact stream (due to the 401) and it is unchanged and should work as before. But as seen in the log excerpt, it is also failing while checking for a public package and we are not maintaining upstream in our private artifact feed but fetching public packages directly from nuget instead.

To Reproduce

We are running Dependabot nightly like this:

trigger: none
schedules:
- cron: '0 2 * * *'
  always: true
  displayName: Nightly
  batch: true
  branches:
    include:
    - master

stages:
- stage: dependabot
  displayName: dependabot
  pool:
    name: Azure Pipelines
    vmImage: ubuntu-latest
  jobs:
  - job: dependabot
    displayName: Run dependabot
    steps:
    - checkout: self
      fetchDepth: 0
    - ${{ each repo in parameters.repos }}:
      - task: dependabot@1
        displayName: ${{ repo }}
        continueOnError: true
        inputs:
          targetRepositoryName: ${{ repo }}
          azureDevOpsAccessToken: $(System.AccessToken)
          gitHubConnection: 'github.com'
        env:
          AZURE_ARTIFACTS_TOKEN: $(System.AccessToken)

With a git repo in Azure DevOps and this configuration:

version: 2
registries:
  publicis-nuget:
    type: nuget-feed
    url: https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json
    token: PAT:${{AZURE_ARTIFACTS_TOKEN}}
updates:
  - package-ecosystem: nuget
    registries:
    - publicis-nuget
    directory: "/"
    target-branch: dev
    commit-message:
      prefix: fix
      include: scope
    milestone: 44711 # App Package Dependency Refresh

Expected behavior We did not change anything. It should simply run as before.

Screenshots -

Extension (please complete the following information):

• Host: Azure DevOps

Server (please complete the following information): -

Additional context

Starting: Initialize job Agent name: 'Hosted Agent' Agent machine name: 'fv-az625-671' Current agent version: '3.232.1' Operating System Runner Image Runner Image Provisioner Current image version: '20231217.2.0' Agent running as: 'vsts' Prepare build directory. Set build variables. Download all required tasks. Downloading task: dependabot (1.25.613) Downloading task: PublishPipelineMetadata (0.216.0) Checking job knob settings. Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES) Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE) Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT) Knob: ProcessHandlerTelemetry = true Source: $(AZP_75787_ENABLE_COLLECT) Knob: IgnoreVSTSTaskLib = true Source: $(AZP_AGENT_IGNORE_VSTSTASKLIB) Knob: CheckForTaskDeprecation = true Source: $(AZP_AGENT_CHECK_FOR_TASK_DEPRECATION) Finished checking job knob settings. Start tracking orphan processes. Finishing: Initialize job

Starting: PM.CF.DictionarySync.Service

Task : Dependabot Description : Automatically update dependencies and vulnerabilities in your code Version : 1.25.613 Author : Tingle Software Help : For help please visit https://github.com/tinglesoftware/dependabot-azure-devops

/usr/bin/docker run --rm -i -e GITHUB_ACCESS_TOKEN=*** -e DEPENDABOT_PACKAGE_MANAGER=nuget -e DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT=5 -e DEPENDABOT_DIRECTORY=/ -e DEPENDABOT_TARGET_BRANCH=dev -e DEPENDABOT_MILESTONE=44711 -e DEPENDABOT_COMMIT_MESSAGE_OPTIONS={"prefix":"fix","include":"scope"} -e DEPENDABOT_EXTRA_CREDENTIALS=[{"type":"nuget_feed","token":"PAT:","url":"https://pkgs.dev.azure.com/TITAN-Products-Publicis/_packaging/P-OS-Artifacts/nuget/v3/index.json"}] -e DEPENDABOT_FAIL_ON_EXCEPTION=true -e AZURE_ORGANIZATION=TITAN-Products-Publicis -e AZURE_PROJECT=Publicis%20OS -e AZURE_REPOSITORY=PM.CF.DictionarySync.Service -e AZURE_ACCESS_TOKEN= -e AZURE_MERGE_STRATEGY=squash ghcr.io/tinglesoftware/dependabot-updater-nuget:1.25 update_script Unable to find image 'ghcr.io/tinglesoftware/dependabot-updater-nuget:1.25' locally 1.25: Pulling from tinglesoftware/dependabot-updater-nuget [...]

[cyberblast]

hmm not sure why it is using such large fonts in the end. sorry it's unintentional...

[JensSchadron]

Just for reference, this is related to #919

[cyberblast]

It seems to only happen with dotnet/nuget projects. Our frontend/npm repos run fine.

I temporarily set dependabot task to run 1.24 image for now to have everything fully functional again. dockerImageTag: '1.24'

Also, regarding that comment:

I also recognized, that it is suddenly fetching > 1000 nuget dependency files. Where as before it were only like ~90.

I recognized that I only counted log lines, so real number of requests would be half of it I guess... Not sure if it matters a lot.

[davesmits]

Not sure if related but we get tihs error

/usr/local/lib/ruby/3.1.0/openssl/buffering.rb:214:in `sysread_nonblock': SSL_read: unexpected eof while reading (OpenSSL::SSL::SSLError)
	from /usr/local/lib/ruby/3.1.0/openssl/buffering.rb:214:in `read_nonblock'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:209:in `read_nonblock'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:79:in `block in readline'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:70:in `loop'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:70:in `readline'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:73:in `block in parse'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:72:in `loop'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:72:in `parse'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/connection.rb:460:in `response'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/connection.rb:291:in `request'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:98:in `block in fetch_stream'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:90:in `loop'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:90:in `fetch_stream'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:45:in `fetch_nupkg_buffer_from_repository'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:31:in `fetch_nuspec_from_repository'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:17:in `block in fetch_nuspec'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:16:in `each'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:16:in `reduce'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:16:in `fetch_nuspec'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:172:in `fetch_dependencies'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:139:in `fetch_transitive_dependencies_impl'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:134:in `fetch_transitive_dependencies'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:39:in `transitive_dependencies'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:183:in `block in transitive_dependencies_from_packages'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:178:in `each'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:178:in `transitive_dependencies_from_packages'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:172:in `add_transitive_dependencies_from_packages'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:131:in `add_transitive_dependencies'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:104:in `parse_dependencies'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:58:in `dependency_set'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:39:in `block in project_file_dependencies'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:37:in `each'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:37:in `project_file_dependencies'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:25:in `parse'
	from bin/update_script.rb:521:in `<main>'

[mburumaxwell]

As @JensSchadron has stated, this is indeed an issue resulting from the merge of #919 but it was something on hold for a while. See comment. The issue is only affecting NuGet when private registries/feeds are in use. All other scenarios and ecosystems seem to continue working as usual.

The cause

The Microsoft team decided to overhaul the way NuGet works in dependabot so that it supports the new GitHub Advanced Security for Azure DevOps in PR #8179.

How come it does not affects the GitHub-hosted version, you asked? Well, there are two things I can think of:

1. Since the update logic changed, some package updates fail in some repositories we have but they are skipped. As recent as yesterday, the update jobs log on GitHub still shows a warning triangle. It did not show this before that PR was merged.
2. The hosted version injects credentials into the VM running the update while this extension passes the credentials as ENV variables. I have no clue how they do that and whether it is something worth exploring. This happened with PNPM a couple of months ago. See issue. If you have ideas, send them my way.

Solution

First some context:

1. Why did this change? I have no clue but of course a paid product will take precedence in such situations, oops!
2. Why did the release with this major bug? I have known of this problem for some weeks now and decided to wait for someone to fix it. Unfortunately, that hasn't happened. I believe the main reason being that most .NET/NuGet users live in Azure DevOps and use this extension. To ascertain the breadth of the issue (not only us) and to also hint at Microsoft (some of them use this extension) that they need to fix what they broke. Thankfully we have a fallback (below).

Hopefully, this gets fixed in the core repo. The solution is probably similar to that for PNPM and it's open for anyone to give a try.

Secretly, I am hoping that this break would result in Azure DevOps getting native support and this extension being killed. It's better for everyone (or so I think).

Workaround

Before a solution is found, you can do something.

If you are using the server, set the dockerImageTag parameter to 1.24.

If you are using the extension:

- task: dependabot@1
  inputs:
    dockerImageTag: '1.24'
    // your other inputs here ...

This will exclude some new changes such as HTML in the PR body and the fix for #730

I hope this gives the much needed clarity.

[davesmits]

thanks @mburumaxwell ; using 1.24 for now worked.

[mburumaxwell]

Tracking https://github.com/dependabot/dependabot-core/issues/8597 which will possibly be fixed by https://github.com/dependabot/dependabot-core/pull/8679

[mburumaxwell]

I am tracking fixes in the core repo, such as https://github.com/dependabot/dependabot-core/pull/8748, then adding them to #927 for testing purposes. If anyone is interested in helping me test you can check out https://github.com/tinglesoftware/dependabot-azure-devops/pull/927#issuecomment-1888560706 and subscribe to the PR for updates.

[cyberblast]

Not sure, but according to this MS article, the problem could be that for v3 nuget protocol at Azure DevOps Artifact Feed it is not sufficient to assemble a nuget.config containing the PAT (as done here by dependabot-core), but Azure Artifacts Credential Provider needs to be used...

[mburumaxwell]

Not sure, but according to this MS article, the problem could be that for v3 nuget protocol at Azure DevOps Artifact Feed it is not sufficient to assemble a nuget.config containing the PAT (as done here by dependabot-core), but Azure Artifacts Credential Provider needs to be used...

This is an interesting find. Thank you. Seeing that Microsoft (Azure DevOps) guys are the ones that changed the implementation, I hope they can either rollback or fix it.

[cyberblast]

Or we add the nuget plugin to the image and provide it the PAT via env variable as described here.

[mburumaxwell]

Or we add the nuget plugin to the image and provide it the PAT via env variable as described here.

@cyberblast can you contribute a PR for that?

[cyberblast]

Not sure if I can manage to do that.

• I'm not really familiar with the code and have no idea about ruby. I was only browsing the code, trying to understand what it does to find a hint for the issue.
• Also I don't have a local dev/docker environment running to test changes locally. Currently I'm only using the Azure DevOps Dependabot Task in some of our CI/CD pipes, nothing else in that regards. These pipes are used by our developers working on something completely different.
• Also regarding the time effort I'm not sure if it will be possible.

If anybody can support with that, it would be highly appreciated. In case I'm wrong and I can manage to find some time to set something up which seems to work, I'll definitely let you know...

[cyberblast]

I found some piece of dockerfile code which our developers are usually using to create images for our own apps. Maybe this could help somebody when looking into the issue...

ARG PAT
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS '{"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/<org>/_packaging/<feed>/nuget/v3/index.json","username":"docker","password":"'${PAT}'"}]}'

RUN curl --proto "=https" --tlsv1.2 -sSf -L https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh  | sh

COPY ./nuget.config ./

I guess the missing piece would be to assemble the correct endpointCredentials json format from dependabot config and set as VSS_NUGET_EXTERNAL_FEED_ENDPOINTS before resolving dependencies .. or something like that

[JensSchadron]

I found some piece of dockerfile code which our developers are usually using to create images for our own apps. Maybe this could help somebody when looking into the issue...

ARG PAT
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS '{"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/<org>/_packaging/<feed>/nuget/v3/index.json","username":"docker","password":"'${PAT}'"}]}'

RUN curl --proto "=https" --tlsv1.2 -sSf -L https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh  | sh

COPY ./nuget.config ./

I guess the missing piece would be to assemble the correct endpointCredentials json format from dependabot config and set as VSS_NUGET_EXTERNAL_FEED_ENDPOINTS before resolving dependencies .. or something like that

Just linking these here as I think they're related: https://github.com/dependabot/dependabot-core/pull/8927 https://github.com/dependabot/dependabot-core/pull/9004

[mburumaxwell]

May be fixed by https://github.com/dependabot/dependabot-core/pull/8927

[Patrick-3000]

This seems to have fixed the issue for me.

[evgenyvalavin]

Was the fix released? Ho to get it working with the latest dependabot version?

[Patrick-3000]

Not really sure anymore what is happening: the day before yesterday it worked. Yesterday our dependabot pipeline failed with

dotnet build in GetAllPackageDependenciesAsync failed. STDOUT: MSBuild version 17.8.3+195e7f5a3 for .NET`

Today it worked again.

And today I see this error in a different dependabot pipeline of ours, which worked yesterday.

Now I am in the situation that I can`t use dependabot for nuget packages due to this issue, and neither for npm packages due to #729 :worried:

[Blackunknown]

I just updated the image version from 1.24 to 1.26.4 because suddenly we were no longer getting work items linked to the PRs created. However 1.26.4 seems to give an error accessing the private feed we have like so:

/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://*.pkgs.visualstudio.com/_packaging/*.*/nuget/v3/index.json. [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] 0 Warning(s) 371 Error(s)

If anyone has any idea how we can get the private feed to work that would be great. Seems this issue is still actively preventing people from going beyond 1.24.

[Sieberkev]

Downgrading from 1.26.671 -> 1.24 "fixed" our pipeline (Telerik nuget feed gave authentication errors), hoping on a permanent fix :)

[Thulasi225]

I just updated the image version from 1.24 to 1.26.4 because suddenly we were no longer getting work items linked to the PRs created. However 1.26.4 seems to give an error accessing the private feed we have like so:

/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://.pkgs.visualstudio.com/_packaging/.*/nuget/v3/index.json. [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] 0 Warning(s) 371 Error(s)

If anyone has any idea how we can get the private feed to work that would be great. Seems this issue is still actively preventing people from going beyond 1.24.

[Rutix]

@mburumaxwell it's a bit hard to see which issue I should monitor. Is this still broken?

[Blackunknown]

We are actually deleting dependabot because of this issue. Because we can't update so linking work items works again. It was a nice idea to save us some work but in the end leads to more work as it is now

[Rutix]

We are actually deleting dependabot because of this issue. Because we can't update so linking work items works again. It was a nice idea to save us some work but in the end leads to more work as it is now

What updated that broke linking work items? It's still works on my side with 1.24 as imageTag

[mburumaxwell]

@mburumaxwell it's a bit hard to see which issue I should monitor. Is this still broken?

I am not sure if it still is broken. Haven't used this in a while. The challenge with this is how much the NuGet space was changing. It is still not back to what it was before Microsoft overhauled it. On GitHub there are still issues like the whole update failing because of multiple target frameworks in Directory.Build.props or timing out for big repository (I have one with 50 projects in it). Overall, the changes in typings are quite a number so I kept off till it stabilizes to focus on where my time is better spent than chasing my own rail.

We are actually deleting dependabot because of this issue. Because we can't update so linking work items works again. It was a nice idea to save us some work but in the end leads to more work as it is now.

Deleting the use of Dependabot in your repository is one way to go about it. There are better options like contributing a fix, pressurize Microsoft to bring official support, migrate to GitHub, or keep using 1.24.

[rhyskoedijk]

1.24 works fine for me too, including work item linking.

@Blackunknown it might be worth double checking that the user account Dependabot is running as has read access to the project and can read work items as originally I had your same issue but it started working after giving the user account project read access.

Despite the regression in dependabot-core, this project is still huge time saver for me and I really appreciate all your work @mburumaxwell.

[rgrace-puck]

Are there any considerations when pinning the 1.24 version? E.g. is the image going to be retained for the foreseeable future?

[ryangribble]

We just set this up for the first time yesterday (Azure DevOps repos, private Azure DevOps nuget feed) and naturally ran into various of these problems. Pinning back to 1.24 did get things to work at least - thanks for the issues and discussions here for providing that information.

However we then wanted to implement "grouping" of dependabot updates, and the settings didn't seem to be working no matter how we configured them in the dependabot.yml file.

We then discovered that grouped updates for nuget was also broken and only "fixed" recently in upstream dependabot-core here - AFTER the above extension version 1.2.4 (roughly around 1.27 of this extension I believe)... however due to the private feed nuget auth issue it seems there is no workable solution if you want grouping of updates?

So I guess I would say one consideration when pinning to 1.24 is that you wont get "grouping"

It's a bit hard to follow all the open issues etc so Im not sure where to look to understand if this is a known issue and there is a PR with proposed fixes we could help test on our setups. Do we just keep trying latest in our dockerImageTag setting every now and then? Will that automatically inlcude the latest from dependabot-core or is it only the latest of the extension and the version of dependabot is still set somewhere else?

[DaleMckeown]

Today my dependabot pipelines are failing even on 1.24. Can anyone confirm whether theirs are working please?