Duplicate paths in repo metadata for apt repository

[paralllax]

When mirroring the apt repo for focal, synchronizations and attempts to download content fail with a duplicate path error for a specific version of the vagrant package.

Debug output

When syncronizing content, the following error is generated:

Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb

This error https://github.com/pulp/pulp_deb/issues/727 arises when the metadata for the repository attempts to reuse the path reference, and is typically an issue with the upstream repository.

I checked the packages files for the repos, and am having trouble locating the problem path/line. Would y'all be able to take a look? For ease, this is a small script I wrote to check the files. I would expect to see it find two matches, but it doesn't, and am curious if I am looking in the wrong place.

#!/bin/bash

# Define the base URL of the repository
REPO_BASE_URL="https://apt.releases.hashicorp.com"

# Define the list of distributions
DISTRIBUTIONS=("focal" "jammy" "bionic")

# Create an associative array to hold all package paths
declare -A package_paths

# Loop through each distribution
for dist in "${DISTRIBUTIONS[@]}"; do
    echo "Processing distribution: $dist"

    # Fetch the Release file for the distribution
    RELEASE_FILE_URL="$REPO_BASE_URL/dists/$dist/Release"
    # Parse the Release file to find the Packages file paths (gzip, bzip2, or uncompressed)
    PACKAGE_FILES=$(curl -sL "$RELEASE_FILE_URL" | grep -E "main/binary-amd64/Packages" | grep -v temp| awk '{print $3}')

    # Loop through each Packages file URL
    for package_file in $PACKAGE_FILES; do
        # Construct the full URL for the Packages file
        PACKAGE_FILE_URL="$REPO_BASE_URL/dists/$dist/$package_file"
        echo "Fetching package file: $PACKAGE_FILE_URL"
    
        # Determine the compression type based on file extension and process accordingly
        case "$PACKAGE_FILE_URL" in
            *gz)
            curl -s "$PACKAGE_FILE_URL" | gunzip | grep 'vagrant_2.3.6' ;;
            *bz2)
            curl -s "$PACKAGE_FILE_URL" | bzip2 -d | grep 'vagrant_2.3.6';;
            *)
            curl -s "$PACKAGE_FILE_URL" | grep 'vagrant_2.3.6' ;;
            esac
        echo '-----'
    done
done

Expected behavior

When sync'ing content, there should be no errors about duplicate paths

Actual behavior

The sync errors in a duplicate path

Reproduction information

Vagrant version

N/A

Host operating system

N/A

Guest operating system

N/A

Steps to reproduce

1. Create a mirror of https://apt.releases.hashicorp.com and sync content.

Vagrantfile

N/A

[paralllax]

It looks like this may have been resolved. Syncs now occur without issue.

    {
      "message": "Update ReleaseFile units",
      "code": "update.release_file",
      "state": "completed",
      "total": null,
      "done": 1,
      "suffix": null
    },

It looks like sync's started to pass recently, on the 9th. @chrisroberts Do you know if there were any upstream changes around that time?

"pulp_created": "2024-01-09T07:10:30.096139Z",

It looks like the metadata files were updated on the 8th

Modify: 2024-01-08 15:10:35.000000000 -0600

If a change was pushed out to fix this, I am happy to close it out. Thank you!

[paralllax]

Hi @chrisroberts,

We started seeing these issues once more a few days back. Can y'all check once more?

2024-02-12T07:53:55.300609Z
Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb

2024-02-11T06:56:17.815189Z
Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb

2024-02-10T07:22:40.293177Z
Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb

2024-02-09T08:12:28.735167Z
Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb

2024-02-08T07:42:02.167189Z
Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb

2024-02-07T07:45:04.578755Z
Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb

2024-02-06T08:05:32.345646Z
Repository version errors : Path errors found. Paths are duplicated: pool/amd64/main/vagrant_2.3.6-1_amd64.deb