dependabot-azure-devops icon indicating copy to clipboard operation
dependabot-azure-devops copied to clipboard

Published 20 hours ago • verified publisher icon tinglesoftware

Test for Nuget auth fix

Open mburumaxwell opened this issue 5 months ago • 23 comments

mburumaxwell avatar Jan 11 '24 02:01 mburumaxwell

Pushed a new image. To test:

- task: dependabot@1
  inputs:
    dockerImageTag: '1.25.2-pullrequest0927-0005'
    // your other inputs here ...

mburumaxwell avatar Jan 12 '24 07:01 mburumaxwell

Hi, I tested dockerImageTag: '1.25.2-pullrequest0927-0005'. Not running well.

  1. "Parsing dependencies information" takes 5 minutes alone. Under 1.24 the whole dependabot process takes 32s for the same repo.
  2. Also then stuck forever (~20m) at
Finding updated dependencies for AutoMapper.Collection.
🌍 --> GET https://api.nuget.org/v3-flatcontainer/automapper.collection/9.0.0/automapper.collection.nuspec
🌍 <-- 200 https://api.nuget.org/v3-flatcontainer/automapper.collection/9.0.0/automapper.collection.nuspec
🌍 --> GET https://api.nuget.org/v3-flatcontainer/automapper/12.0.0/automapper.nuspec
🌍 <-- 200 https://api.nuget.org/v3-flatcontainer/automapper/12.0.0/automapper.nuspec
Updating AutoMapper.Collection from 7.0.1 to 
running NuGet updater:
/opt/nuget/NuGetUpdater/NuGetUpdater.Cli update --repo-root /home/dependabot/dependabot-updater/tmp/<devops-collection>/<devops-project>/_git/PM.DCP.API.Service --solution-or-project /home/dependabot/dependabot-updater/tmp/<devops-collection>/<devops-project>/_git/PM.DCP.API.Service/src/PM.DCP.Service.Application/PM.DCP.Service.Application.csproj --dependency AutoMapper.Collection --new-version 9.0.0 --previous-version 7.0.1  --verbose

then eventually

[...]
/tmp/package-dependency-resolution_i3u82p/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/<devops-collection>/_packaging/P-OS-Artifacts/nuget/v3/index.json.
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/<devops-collection>/_packaging/P-OS-Artifacts/nuget/v3/index.json. [/tmp/package-dependency-resolution_i3u82p/Project.csproj]
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error :   Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_i3u82p/Project.csproj]
[...]
STDERR: 
Package [AutoMapper.Collection] Does not exist as a dependency in [/home/dependabot/dependabot-updater/tmp/<devops-collection>/<devops-project>/_git/PM.DCP.API.Service/src/PM.DCP.Service.Application/PM.DCP.Service.Application.csproj].

The strange thing here is, that the package is not contained in the private feed at all. We don't use any upstream but load public packages always directly from public source (nuget.org). Only our own packages are available from the private feed. So I totally don't get what it is doing at all... Package AutoMapper.Collection needs to be loaded from nuget.org, not the private feed at all.

cyberblast avatar Jan 12 '24 08:01 cyberblast

Will give it some tries over the weekend.

JensSchadron avatar Jan 12 '24 10:01 JensSchadron

I just tried but no luck for me neither

/usr/local/lib/ruby/3.1.0/openssl/buffering.rb:214:in `sysread_nonblock': SSL_read: unexpected eof while reading (OpenSSL::SSL::SSLError) (Excon::Error::Socket)
	from /usr/local/lib/ruby/3.1.0/openssl/buffering.rb:214:in `read_nonblock'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:209:in `read_nonblock'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:79:in `block in readline'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:70:in `loop'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:70:in `readline'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:73:in `block in parse'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:72:in `loop'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:72:in `parse'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/connection.rb:460:in `response'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/connection.rb:291:in `request'

so 'same` as here https://github.com/tinglesoftware/dependabot-azure-devops/issues/921#issuecomment-1878765220

davesmits avatar Jan 12 '24 15:01 davesmits

Gave it a test run a little while ago and my result is basically the same, that it doesn't work with private registries (Azure Artifacts in my case)

Also... I tried comparing the logs between 1.24 and 1.25.2-pullrequest0927-0005 and noticed that it doesn't even check the private feed (seemingly) on the newer version.

1.24:

Checking if {packageName} 1.0.2 needs updating
🌍 --> GET https://pkgs.dev.azure.com/{organisationName}/{projectName}/_packaging/{feedName}/nuget/v3/index.json
🌍 <-- 200 https://pkgs.dev.azure.com/{organisationName}/{projectName}/_packaging/{feedName}/nuget/v3/index.json
🌍 --> GET https://pkgs.dev.azure.com/{organisationName}/2110eeea-1457-4552-8bbd-5d9d2a0a4338/_packaging/b051a3cf-2486-4e06-8810-1e4e6beabb2b/nuget/v3/flat2/{packageName}/index.json
🌍 <-- 200 https://pkgs.dev.azure.com/{organisationName}/2110eeea-1457-4552-8bbd-5d9d2a0a4338/_packaging/b051a3cf-2486-4e06-8810-1e4e6beabb2b/nuget/v3/flat2/{packageName}/index.json
🌍 --> GET https://azuresearch-usnc.nuget.org/query?q={packageName}&prerelease=true&semVerLevel=2.0.0
🌍 <-- 200 https://azuresearch-usnc.nuget.org/query?q={packageName}&prerelease=true&semVerLevel=2.0.0
Requirements to unlock own
Requirements update strategy 
Updating {packageName} from 1.0.2 to 1.2.0
Submitting {packageName} pull request for creation.

1.25.2-pullrequest0927-0005:

Checking if {packageName} 1.0.2 needs updating
🌍 --> GET https://api.nuget.org/v3/registration5-gz-semver2/{packageName}/index.json
🌍 <-- 404 https://api.nuget.org/v3/registration5-gz-semver2/{packageName}/index.json
🌍 --> GET https://api.nuget.org/v3-flatcontainer/{packageName}/1.0.2/{packageName}.nuspec
🌍 <-- 404 https://api.nuget.org/v3-flatcontainer/{packageName}/1.0.2/{packageName}.nuspec
Requirements to unlock all
Requirements update strategy 
Finding updated dependencies for {packageName}.
🌍 --> GET https://api.nuget.org/v3-flatcontainer/{packageName}/1.2.0/{packageName}.nuspec
🌍 <-- 404 https://api.nuget.org/v3-flatcontainer/{packageName}/1.2.0/{packageName}.nuspec
Updating {packageName} from 1.0.2 to 
running NuGet updater:
/opt/nuget/NuGetUpdater/NuGetUpdater.Cli update --repo-root /home/dependabot/dependabot-updater/tmp/{organisationName}/DemoProject-Internal/_git/dependabot-github-919 --solution-or-project /home/dependabot/dependabot-updater/tmp/{organisationName}/DemoProject-Internal/_git/dependabot-github-919/source/DependabotRepro.ClassLibrary/DependabotRepro.ClassLibrary.csproj --dependency {packageName} --new-version 1.2.0 --previous-version 1.0.2  --verbose
  No global.json files found.
  No dotnet-tools.json files found.
Running for project [/home/dependabot/dependabot-updater/tmp/{organisationName}/DemoProject-Internal/_git/dependabot-github-919/source/DependabotRepro.ClassLibrary/DependabotRepro.ClassLibrary.csproj]
  Running for SDK-style project
dotnet build in GetAllPackageDependenciesAsync failed. STDOUT: MSBuild version 17.8.3+195e7f5a3 for .NET
  Determining projects to restore...
/tmp/package-dependency-resolution_CSEqxR/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/{organisationName}/{projectName}/_packaging/{feedName}/nuget/v3/index.json.
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/{organisationName}/{projectName}/_packaging/{feedName}/nuget/v3/index.json. [/tmp/package-dependency-resolution_CSEqxR/Project.csproj]
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error :   Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_CSEqxR/Project.csproj]

Build FAILED.

and shortly after failing with the following stacktrace:

/home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11188/lib/types/_types.rb:222:in `must': Passed `nil` into T.must (TypeError)
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-1bb15b002f29/common/lib/dependabot/pull_request_creator/branch_namer/solo_strategy.rb:41:in `prefixes'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-1bb15b002f29/common/lib/dependabot/pull_request_creator/branch_namer/solo_strategy.rb:32:in `new_branch_name'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-1bb15b002f29/common/lib/dependabot/pull_request_creator/branch_namer.rb:30:in `new_branch_name'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-1bb15b002f29/common/lib/dependabot/pull_request_creator.rb:305:in `azure_creator'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11188/lib/types/private/methods/call_validation.rb:272:in `bind_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11188/lib/types/private/methods/call_validation.rb:272:in `validate_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11188/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-1bb15b002f29/common/lib/dependabot/pull_request_creator.rb:235:in `create'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11188/lib/types/private/methods/call_validation.rb:272:in `bind_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11188/lib/types/private/methods/call_validation.rb:272:in `validate_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11188/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
	from bin/update_script.rb:823:in `block in <main>'
	from bin/update_script.rb:539:in `each'
	from bin/update_script.rb:539:in `<main>'

JensSchadron avatar Jan 14 '24 23:01 JensSchadron

Thanks @cyberblast @davesmits , @JensSchadron for reporting. I am also testing this and getting the same results. I will keep updating to the latest commit from the code repo to see if there are improvements about once a week. New tag version is 1.25.2-pullrequest0927-0011

- task: dependabot@1
  inputs:
    dockerImageTag: '1.25.2-pullrequest0927-0011'
    // your other inputs here ...

Still seeing the same issue with this one but it may behave differently on another setup.

mburumaxwell avatar Jan 19 '24 05:01 mburumaxwell

Reports that the latest bits are working with private feeds but I still can't get it to work. Maybe someone else can?

- task: dependabot@1
  inputs:
    dockerImageTag: '1.25.4-pullrequest0927-0002'
    // your other inputs here ...

mburumaxwell avatar Jan 24 '24 06:01 mburumaxwell

The SSL error is at least gone.

dotnet build in GetAllPackageDependenciesAsync failed. STDOUT: MSBuild version 17.8.3+195e7f5a3 for .NET
  Determining projects to restore...
/tmp/package-dependency-resolution_wbmdXd/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/pandoraintelligence/_packaging/Pandora.Platform/nuget/v3/index.json.
/tmp/package-dependency-resolution_wbmdXd/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/pandoraintelligence/_packaging/Pandora.Platform/nuget/v3/index.json.
/tmp/package-dependency-resolution_wbmdXd/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/pandoraintelligence/_packaging/Pandora.Platform/nuget/v3/index.json.
/tmp/package-dependency-resolution_wbmdXd/Project.csproj : error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/pandoraintelligence/_packaging/Pandora.Platform/nuget/v3/index.json.
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/pandoraintelligence/_packaging/Pandora.Platform/nuget/v3/index.json. [/tmp/package-dependency-resolution_wbmdXd/Project.csproj]
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error :   Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_wbmdXd/Project.csproj]

davesmits avatar Jan 24 '24 08:01 davesmits

I gave it a quick shot but seems to behave the same as before :(

cyberblast avatar Jan 24 '24 08:01 cyberblast

Some more fixes have been put in. The new version for testing is 1.25.4-pullrequest0927-0011. Still having issues with private feeds. Maybe someone else will have better luck.

- task: dependabot@1
  inputs:
    dockerImageTag: '1.25.4-pullrequest0927-0011'
    // your other inputs here ...

mburumaxwell avatar Jan 29 '24 15:01 mburumaxwell

As of today:

- task: dependabot@1
  inputs:
    dockerImageTag: '1.26.1-pullrequest0927-0038'
    // your other inputs (if any) go here ...

mburumaxwell avatar Feb 05 '24 06:02 mburumaxwell

Hey. I gave it a try (1.26.1-pullrequest0927-0038) but get some errors...

running NuGet updater:
/opt/nuget/NuGetUpdater/NuGetUpdater.Cli update --repo-root /home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo> --solution-or-project /home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/tests/<some-proj>/<some-proj>.csproj --dependency <some-private-package> --new-version 4.2.0 --previous-version 3.4.0 --verbose
  No dotnet-tools.json files found.
Running for project [/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/tests/<some-proj>/<some-proj>.csproj]
  Running for SDK-style project
    Package [<some-private-package>] Does not exist as a dependency in [/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/tests/<some-proj>/<some-proj>.csproj].
Update complete.
/home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/configuration.rb:296:in `call_validation_error_handler_default': Parameter 'milestone': Expected type T.nilable(T.any(Integer, T::Array[String])), got type String with value "44711" (TypeError)
Caller: /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/call_validation.rb:215
Definition: /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-df3e6e1cc9f7/common/lib/dependabot/pull_request_creator.rb:174
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/configuration.rb:303:in `call_validation_error_handler'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/call_validation.rb:300:in `report_error'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/call_validation.rb:218:in `block in validate_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/signature.rb:235:in `block in each_args_value_type'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/signature.rb:229:in `each'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/signature.rb:229:in `each_args_value_type'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/call_validation.rb:215:in `validate_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11226/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
	from bin/update_script.rb:801:in `new'
	from bin/update_script.rb:801:in `block in <main>'
	from bin/update_script.rb:540:in `each'
	from bin/update_script.rb:540:in `<main>'
##[error]The process '/usr/bin/docker' failed with exit code 1

running NuGet updater:
/opt/nuget/NuGetUpdater/NuGetUpdater.Cli update --repo-root /home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo> --solution-or-project /home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/test/<some-proj>/<some-proj>.csproj --dependency Microsoft.Azure.WebJobs.Extensions.EventHubs --new-version 6.0.2 --previous-version 5.4.0 --verbose
  No dotnet-tools.json files found.
Running for project [/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/test/<some-proj>/<some-proj>.csproj]
  Running for SDK-style project
    Package [Microsoft.Azure.WebJobs.Extensions.EventHubs] Does not exist as a dependency in [/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/home/dependabot/dependabot-updater/tmp/<some-path>/_git/<some-repo>/test/<some-proj>/<some-proj>.csproj].
Update complete.
bin/update_script.rb:747:in `block (2 levels) in <main>': undefined method `directory' for nil:NilClass (NoMethodError)

      next if updated_files.first.directory != "/" && !title.end_with?(" in #{updated_files.first.directory}")
                                 ^^^^^^^^^^
	from bin/update_script.rb:724:in `each'
	from bin/update_script.rb:724:in `block in <main>'
	from bin/update_script.rb:540:in `each'
	from bin/update_script.rb:540:in `<main>'
##[error]The process '/usr/bin/docker' failed with exit code 1

I also recognized, that it is downloading the whole https://api.nuget.org/v3-flatcontainer nuspecs for every dependabot task again and again :/ we are running ~ 90 dependabot tasks in our nightly pipe (one per repo)

cyberblast avatar Feb 05 '24 12:02 cyberblast

@cyberblast thanks for trying this one out. Unfortunately, the changes in the upstream repo are still ongoing. For the jobs you are running at night, I recommend using 1.24 until we can tell what is going on. I haven't had much time to look through the code to find a solution so hopefully it gets fixed in the upstream or a community fix.

mburumaxwell avatar Feb 05 '24 12:02 mburumaxwell

As of today:

- task: dependabot@1
  inputs:
    dockerImageTag: '1.26.1-pullrequest0927-0043'
    // your other inputs (if any) go here ...

mburumaxwell avatar Feb 12 '24 05:02 mburumaxwell

As of today:

- task: dependabot@1
  inputs:
    dockerImageTag: '1.26.4-pullrequest0927-0002'
    // your other inputs (if any) go here ...

mburumaxwell avatar Feb 19 '24 16:02 mburumaxwell

Updated to the mentioned version, but the issue persists.

JensSchadron avatar Feb 22 '24 05:02 JensSchadron

I did the try 1.26.1-pullrequest0927-0043 version and seems the SSL errors / Http Errors we had before are gone

now got this stacktrace (just lack of ruby knowledge to do something with it)

    Package [Polly] Does not exist as a dependency in [/home/dependabot/dependabot-updater/tmp/pandoraintelligence/Pandora\%20Box/_git/Pandora.NuGet/home/dependabot/dependabot-updater/tmp/pandoraintelligence/Pandora\%20Box/_git/Pandora.NuGet/Pandora.Nats.IntegrationTests/Pandora.Nats.IntegrationTests.csproj].
Update complete.
bin/update_script.rb:756:in `block (2 levels) in <main>': undefined method `directory' for nil:NilClass (NoMethodError)

      next if updated_files.first.directory != "/" && !title.end_with?(" in #{updated_files.first.directory}")
                                 ^^^^^^^^^^
	from bin/update_script.rb:733:in `each'
	from bin/update_script.rb:733:in `block in <main>'
	from bin/update_script.rb:549:in `each'
	from bin/update_script.rb:549:in `<main>'

davesmits avatar Mar 12 '24 11:03 davesmits

As of today:

- task: dependabot@1
  inputs:
    dockerImageTag: '1.27.2-pullrequest0927-0002'
    // your other inputs (if any) go here ...

This may still not work as we wait for https://github.com/dependabot/dependabot-core/pull/8927

mburumaxwell avatar Mar 18 '24 05:03 mburumaxwell

As of today:

- task: dependabot@1
  inputs:
    dockerImageTag: '1.27.4-pullrequest0927-0002'
    // your other inputs (if any) go here ...

mburumaxwell avatar Apr 08 '24 05:04 mburumaxwell

Worked for me

davesmits avatar Apr 09 '24 09:04 davesmits

oke found new minor issue; workitem linking is now broken

davesmits avatar Apr 09 '24 11:04 davesmits

Did the latest changes from dependabot-core #1090 make improvements to fixing the issue?

james-asebp avatar Apr 15 '24 17:04 james-asebp

Updated to the mentioned version, but the issue persists.

Thulasi225 avatar Apr 24 '24 09:04 Thulasi225