Replacement for `git://` URLs does not seem to be working for submodules

[sschuberth]

Despite 3d27d61542495d8b51848861652bf1b31ef82e49, the provenance resolver that runs as part of the scanner does not seem to be able to clone Git repositories that contain submodules which still refer via git://´ to GitHub repositories (GitHub has deprecated the git://` protocol):

Could not resolve nested provenance for package 'PyPI::scikit-image:0.19.3': IOException: Running 'git submodule update --recursive' in '/tmp/ort-DefaultWorkingTreeCache10066802054424201727' failed with exit code 1:
Cloning into '/tmp/ort-DefaultWorkingTreeCache10066802054424201727/doc/tools/gitwash'...
fatal: unable to connect to github.com:
github.com[0: 140.82.121.3]: errno=Connection timed out

fatal: clone of 'git://github.com/matthew-brett/gitwash.git' into submodule path '/tmp/ort-DefaultWorkingTreeCache10066802054424201727/doc/tools/gitwash' failed
Failed to clone 'doc/tools/gitwash'. Retry scheduled
Cloning into '/tmp/ort-DefaultWorkingTreeCache10066802054424201727/doc/tools/gitwash'...
fatal: unable to connect to github.com:
github.com[0: 140.82.121.4]: errno=Connection timed out

fatal: clone of 'git://github.com/matthew-brett/gitwash.git' into submodule path '/tmp/ort-DefaultWorkingTreeCache10066802054424201727/doc/tools/gitwash' failed
Failed to clone 'doc/tools/gitwash' a second time, aborting

YAML entry:

  - id: "PyPI::scikit-image:0.19.3"
    package_provenance:
      vcs_info:
        type: "Git"
        url: "https://github.com/scikit-image/scikit-image.git"
        revision: ""
        path: ""
      resolved_revision: "c1af2bf80d2c8845bbbf06222f87e8b58a8e5c1e"
    nested_provenance_resolution_issue:
      timestamp: "2024-07-18T12:44:39.719752127Z"
      source: "scanner"
      message: "Could not resolve nested provenance for package 'PyPI::scikit-image:0.19.3':\
        \ IOException: Running 'git submodule update --recursive' in '/tmp/ort-DefaultWorkingTreeCache9932109886528191008'\
        \ failed with exit code 1:\nCloning into '/tmp/ort-DefaultWorkingTreeCache9932109886528191008/doc/tools/gitwash'...\n\
        fatal: unable to connect to github.com:\ngithub.com[0: 140.82.121.3]: errno=Connection\
        \ timed out\n\nfatal: clone of 'git://github.com/matthew-brett/gitwash.git'\
        \ into submodule path '/tmp/ort-DefaultWorkingTreeCache9932109886528191008/doc/tools/gitwash'\
        \ failed\nFailed to clone 'doc/tools/gitwash'. Retry scheduled\nCloning into\
        \ '/tmp/ort-DefaultWorkingTreeCache9932109886528191008/doc/tools/gitwash'...\n\
        fatal: unable to connect to github.com:\ngithub.com[0: 140.82.121.3]: errno=Connection\
        \ timed out\n\nfatal: clone of 'git://github.com/matthew-brett/gitwash.git'\
        \ into submodule path '/tmp/ort-DefaultWorkingTreeCache9932109886528191008/doc/tools/gitwash'\
        \ failed\nFailed to clone 'doc/tools/gitwash' a second time, aborting\n"
      severity: "ERROR"

However, the more severe issue seems to be that as a result of the above, the PyPI::scikit-image:0.19.3 is not scanned at all, neither as a repository without submodules, nor the source artifact (which was found). At least there is no corresponding provenance entry in the scan_results section.

[vw-anton]

Wouldn't it suffice if we enforce this on a system level when building the Docker image via sudo git config --system url."https://".insteadOf git:// ?

[sschuberth]

I guess that should work.

[fviernau]

Wouldn't it suffice if we enforce this on a system level when building the Docker image via sudo git config --system url."https://".insteadOf git:// ?

I guess that should work.

Would the analyzer result contain git:// URLs then?

[sschuberth]

Would the analyzer result contain git:// URLs then?

I believe so, yes.

[cz-dev-ge]

Here is a .NET minimal example:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <RootNamespace>antlrl3_test</RootNamespace>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  
  <ItemGroup>
    <PackageReference Include="Antlr3.Runtime" Version="3.5.1" />
  </ItemGroup>

</Project>

put it into a antlr3-test.csproj within an empty folder and scan it.

[sschuberth]

For testing, also see https://github.com/wix/Detox/issues/4871, maybe.

[sschuberth]

For testing, also see wix/Detox#4871, maybe.

For the record, I just verified that plain git CLI is able to recursively clone that repo when configuring git config --global url."https://github.com/".insteadOf "[email protected]:".