spacewalk-repo-sync only works for Debian repositories with public listings allowed

[stdevel]

Problem description

When synchronizing Debian repositories, spacewalk-repo-sync fails if the web server hosting the repository blocks directory listing:

# spacewalk-repo-sync -c cashicorp-debian-11 -vvv
11:54:24 ======================================
11:54:24 | Channel: cashicorp-debian-11
11:54:24 ======================================
11:54:24 Sync of channel started.
ERROR: Download of package list failed.
11:54:25 Repo URL: https://apt.releases.hashicorp.com/bookworm
...

Try accessing the URL https://apt.releases.hashicorp.com/bookworm using browser, it won't work.

Using a repository with directory listing enabled works like a charm, e.g. https://download.docker.com/linux/debian/dists/bullseye/stable/binary-amd64/.

Steps to reproduce

1. Create a repository with the URL https://apt.releases.hashicorp.com/bookworm or https://releases.mondoo.com/debian
2. Try to sync the repository using spacewalk-repo-sync
3. Watch the error message

Uyuni version

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : uyuni-server-stable
Name           : Uyuni-Server-release
Version        : 2024.02-230900.213.1.uyuni3
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Installed Size : 1.4 KiB
Installed      : Yes (automatically)
Status         : up-to-date
Source package : Uyuni-Server-release-2024.02-230900.213.1.uyuni3.src
Summary        : Uyuni Server
Description    : 
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Uyuni proxy version (if used)

No response

Useful logs

/var/log/rhn/rhn_taskomatic_daemon.log:2024-03-06 11:55:00,045 [Thread-2007] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating new DEB repository for channel cashicorp-debian-11
/var/log/rhn/rhn_taskomatic_daemon.log:2024-03-06 11:55:00,049 [Thread-2007] WARN  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel metadata signing is disabled. APT repository cashicorp-debian-11 is not secure.Refer to the Debian apt-secure manpage.
/var/log/rhn/rhn_taskomatic_daemon.log:2024-03-06 11:55:00,050 [Thread-2007] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository metadata generation for 'cashicorp-debian-11' finished in 0 seconds

Additional information

Looks like the appropriate spacewalk-repo-sync module needs to iterate through available folders: https://github.com/uyuni-project/uyuni/blob/01909a02825537835a1299cda0de1e81ae5e4a54/python/spacewalk/satellite_tools/repo_plugins/deb_src.py#L267

I'm not a Debian package maintainer, but I think the following files of a repository are of major interest - so maybe they could be accessed directy instead of iterating?

• InRelease
• Packages
• Packages.bz2
• Packages.gz
• Release
• Release.gpg

[agraul]

There are different Debian repository layouts, the hashicorp one is not flat and needs you to add uyuni_suite, uyuni_component, and uyuni_arch to the URL. There are examples on https://www.uyuni-project.org/uyuni-docs/en/uyuni/administration/custom-channels.html.

Based on https://www.hashicorp.com/official-packaging-guide, maybe https://apt.releases.hashicorp.com?uyuni_suite=bookworm&uyuni_component=main&uyuni_arch=amd64 works

BTW, data.iter_content is just downloading a file in chunks

>>> data = requests.get("https://download.docker.com/linux/debian/dists/bullseye/stable/binary-amd64/Packages.gz")
>>> di = data.iter_content(chunk_size=8)
>>> next(di)
b'\x1f\x8b\x08\x00\x98s!f'
>>> next(di)
b'\x00\x03\xec\x9b]o\x9c9'

[avshiliaev]

Closing as solved, feel free to reopen if needed :smiley: