aptly-dev
Inconsistency between REST API description and CLI description of the same mirror, db files are missing.
Detailed Description
Found inconsistency between cli output and REST API output:
aptly mirror show ubuntu-jammy-updates-full
Name: ubuntu-jammy-updates-full
Archive Root URL: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/
Distribution: jammy-updates
Components: main, restricted, universe, multiverse
Architectures: i386, amd64
Download Sources: no
Download .udebs: no
Last update: 2024-09-10 23:42:22 UTC
Number of packages: 31724
Information from release file:
Acquire-By-Hash: yes
Architectures: amd64 arm64 armhf i386 ppc64el riscv64 s390x
Codename: jammy
Components: main restricted universe multiverse
Date: Tue, 10 Sep 2024 22:20:16 UTC
Description: Ubuntu Jammy Updates
Label: Ubuntu
Origin: Ubuntu
Suite: jammy-updates
Version: 22.04
curl http://localhost:8080/api/mirrors/ubuntu-jammy-updates-full | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 832 100 832 0 0 99059 0 --:--:-- --:--:-- --:--:-- 101k
{
"UUID": "e267beaa-f65e-4c16-89d0-5413d368e560",
"Name": "ubuntu-jammy-updates-full",
"ArchiveRoot": "http://us-east-1.ec2.archive.ubuntu.com/ubuntu/",
"Distribution": "jammy-updates",
"Components": [
"main",
"restricted",
"universe",
"multiverse"
],
"Architectures": [
"i386",
"amd64"
],
"Meta": {
"Acquire-By-Hash": "yes",
"Architectures": "amd64 arm64 armhf i386 ppc64el riscv64 s390x",
"Codename": "jammy",
"Components": "main restricted universe multiverse",
"Date": "Thu, 08 Aug 2024 14:56:37 UTC",
"Description": " Ubuntu Jammy Updates\n",
"Label": "Ubuntu",
"Origin": "Ubuntu",
"Suite": "jammy-updates",
"Version": "22.04"
},
"LastDownloadDate": "2024-08-09T02:42:42.564974851Z",
"Filter": "",
"Status": 0,
"WorkerPID": 0,
"FilterWithDeps": false,
"SkipComponentCheck": false,
"SkipArchitectureCheck": false,
"DownloadSources": false,
"DownloadUdebs": false,
"DownloadInstaller": false
}
Also, the mirror update does not work for this mirror:
│ [GIN] 2024/09/11 - 00:07:42 | 200 | 4.479296ms | 127.0.0.1 | GET "/api/mirrors/ubuntu-jammy-updates-full" │
│ 2024/09/11 00:07:43 ubuntu-jammy-updates-full: Starting mirror update │
│ 2024/09/11 00:07:43 Executing task synchronously │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease │
│ gpgv: can't allocate lock for 'key.gpg' │
│ gpgv: Signature made Tue Sep 10 23:23:19 2024 UTC │
│ gpgv: using RSA key F6ECB3762474EDA9D21B7022871920D1991BC93C │
│ gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2018) <[email protected]>" │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-i386/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-i386/Packages.gz │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-amd64/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-amd64/Packages.gz │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-i386/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-i386/Packages.gz │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-amd64/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-amd64/Packages.gz │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-i386/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-i386/Packages.gz │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-amd64/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-amd64/Packages.gz │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-i386/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-i386/Packages.gz │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-amd64/Packages.gz... │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-amd64/Packages.gz │
│ [GIN] 2024/09/11 - 00:07:44 | 500 | 979.47889ms | 127.0.0.1 | PUT "/api/mirrors/ubuntu-jammy-updates-full" │
│ Error #01: unable to update: open /root/aptly/storage/db/060118.sst: no such file or directory
Note the last line: error locating part of the DB. I tried to cleanup and recover, but the error is the same, part of DB is missing.
Aptly running in the docker container, most operations are done using cli, and some operations are done with REST API. I also have db backups if this will help.
I do not have particular steps to reproduce, since I do not know exactly when this inconsistency started. Please let me know how to approach this problem. Mirror re-creation is an option, but I would like to avoid it if possible.
To the best of my understanding, the issue is caused by opening db during the mirror update on this line.
Context
I want consistent data about the same object returned via REST API and CLI.
Possible Implementation
Your Environment
Aptly 1.5.0 Docker container with mounted volume for data.
@cheeeee If you would be willing to attach your database backups here while you have them, they could potentially be of use in troubleshooting. Though without a log of commands run against the aptly it would be difficult to pinpoint exactly what caused the corruption, if anything.
You mentioned this is running in a container, was the container potentially killed by a docker rm -f or a pod rollover? I could see SIGKILL potentially causing some db corruption if something wasn't written to disk.
Tbh we are just running high-level goleveldb API commands here and aren't well equipped to troubleshoot the database corruption. Maybe upstream leveldb repos could offer more support?
if this is an option, you could try the CI builds. A lot of race conditions and database issues were fixed...
does the inconsistency persist over aptly restarts ?
aptly 1.6.0 has been released https://github.com/aptly-dev/aptly/discussions/1414 which includes many bug fixes compared to 1.5.0.
Could you give it a try ?
We believe the issue is fixed in latest aptly, closing. Please reopen if the issue persists !