packages missing after concurrent publish

[flixr]

We are still seeing the issue #1125 with concurrent publish once in a while with 1.6.0:

E.g. after uploading (to different tmp folders) and publishing two packages in parallel:

[GIN] 2025/04/09 - 16:55:59 | 200 |    8.029972ms |       10.0.0.42 | POST     "/api/files/tmp.cAIGfOwQt99ukdk"
[GIN] 2025/04/09 - 16:55:59 | 200 |    7.242806ms |       10.0.0.45 | POST     "/api/files/tmp.zO8MZsfCgL7u2Uw"
[GIN] 2025/04/09 - 16:55:59 | 200 |   94.609874ms |       10.0.0.45 | POST     "/api/repos/roboception-jammy-testing/file/tmp.zO8MZsfCgL7u2Uw/foobar_0.21.0+11+g1f08260-0+20250409+164139~jammy_arm64.deb"
[GIN] 2025/04/09 - 16:55:59 | 200 |  228.518648ms |       10.0.0.42 | POST     "/api/repos/roboception-focal-testing/file/tmp.cAIGfOwQt99ukdk/foobar_0.21.0+11+g1f08260-0+20250409+164136~focal_arm64.deb"
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:
[GIN] 2025/04/09 - 16:56:05 | 200 |  5.673881188s |       10.0.0.45 | PUT      "/api/publish/testing/jammy"
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:
[GIN] 2025/04/09 - 16:56:09 | 200 |  9.535539358s |       10.0.0.42 | PUT      "/api/publish/testing/focal"

Afterwards foobar_0.21.0+11+g1f08260-0+20250409+164136~focal_arm64.deb is listed as a package, but trying to download it fails:

E: Failed to fetch http://our-apt-server.com/testing/pool/main/f/foobar/foobar_0.21.0+11+g1f08260-0+20250409+164136~focal_arm64.deb  404  Not Found [IP: 10.0.0.1 80]

The file is actually there in ./pool/e4/be/af861880689cd3d9b8384504f350_foobar_0.21.0+11+g1f08260-0+20250409+164136~focal_arm64.deb but the file ./public/testing/pool/main/f/foobar/foobar_0.21.0+11+g1f08260-0+20250409+164136~focal_arm64.deb did NOT exist as it should be (and is the case for the jammy file).

So we triggered our CI job again, uploading a new package

[GIN] 2025/04/10 - 08:58:53 | 200 |    8.154728ms |       10.0.0.45 | POST     "/api/files/tmp.SBrA1KDt6obBSsJ"
[GIN] 2025/04/10 - 08:58:54 | 200 |  293.746575ms |       10.0.0.45 | POST     "/api/repos/roboception-focal-testing/file/tmp.SBrA1KDt6obBSsJ/foobar_0.21.0+11+g1f08260-0+20250410+084431~focal_arm64.deb"
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:
[GIN] 2025/04/10 - 08:59:05 | 200 | 11.466402033s |       10.0.0.45 | PUT      "/api/publish/testing/focal"

And afterwards both files were correctly linked to the published repo/pool:

./public/testing/pool/main/f/foobar/foobar_0.21.0+11+g1f08260-0+20250410+084431~focal_arm64.deb
./public/testing/pool/main/f/foobar/foobar_0.21.0+11+g1f08260-0+20250409+164136~focal_arm64.deb

[russelltg]

I've seen this a few times before as well

[neolynx]

Thanks for reporting.

I have also seen such behavior in certain situations. I think this will need a test setup to reproduce and debug. I suspect there might be some issues with goleveldb transactions and locking.

[flixr]

This is super annoying and regularly results in CI failures here and we have to manually publish the repo again so that the last packages are actually available.

Any updates here? Anything we can help with to debug this further?

[agustinhenze]

Can I have some review over the linked PR? Also for someone of the reporters, would you mind to test this change to see if the issue keeps happening (pretty confident that it shouldn't, but having some confirmation would be much better).