temurin-11-jre and temurin-17-jdk Debian package aren't served properly on JFrog

[hdurand-work]

Hello,

Same as #1002 (but it was closed)

Since yesterday packages.adoptium.net stopped serving some of the temurin's Debian package.

debmirror's log:

Errors:
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-11/temurin-11-jre_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
Failed to download files (3 errors)

Our firewall doesn't seem to see any problem:

After digging into it, we discovered that the packages aren't visible on JFrog's web UI even though it seems to "know" that they exist

Alpine in comparison:

Debian 11 JRE in comparison:

[karianna]

https://github.com/adoptium/installer/pull/810 to resolve this

[steelhead31]

I also believe these packages should now be being served properly.

[karianna]

@hdurand-artic Can you try again and let us know?

[sxa]

This is odd. I just performed a quick new test on Debian 12 (bookworm) on x64 and 11-jre installed ok but 17-jdk didn't.

Err:1 https://packages.adoptium.net/artifactory/deb bookworm/main amd64 temurin-17-jdk amd64 17.0.10.0.0+7
  Error reading from server. Remote end closed connection [IP: 151.101.131.42 443]
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb  Error reading from server. Remote end closed connection [IP: 151.101.131.42 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
root@f76153b94ba7:/# wget https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb
--2024-01-26 23:28:26--  https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb
Resolving packages.adoptium.net (packages.adoptium.net)... 151.101.67.42, 151.101.195.42, 151.101.3.42, ...
Connecting to packages.adoptium.net (packages.adoptium.net)|151.101.67.42|:443... connected.
HTTP request sent, awaiting response... 200 OK

I tried the install several times with the same results, then paused for a minute or so before trying the wget on the URL, which worked, and then after that the apt install worked. It does feel somehow cache/proxy related.

EDIT: Tried in a new docker container after the above and it worked ok straight away with temurin-17-jdk - maybe my wget did some magic :-)

[hdurand-work]

@karianna Some errors are still here, including on new packages:

Errors:
 Download of pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-11/temurin-11-jre_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-17/temurin-17-jre_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-8/temurin-8-jre_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Failed to download files (8 errors)!

Some of them are indeed broken in JFrog - for example:

but, weirdly enough, not all of them: - for example:

[steelhead31]

I've checked all of these artifacts ( and re-uploaded them, and checked again! ), and it appears all are present and correct, there may be an issue with the jfrog web UI, when clicking on the location page. but for me at least, it then redirects to the correct artifact when browsing, e.g...  https://packages.adoptium.net/ui/repos/tree/General/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.1.0.0+12_amd64.deb   .. Im going to log the issue with the webui with jfrog.. but I've also tested that apt install works for JDKs & JREs for all version and installs the current/latest versions..

[hdurand-work]

@steelhead31

I am not sure it's working... it doesn't seem to error out on download, but it does say that there were timeouts... which is weird:

Files to download: 568 MiB
[  0%] Getting: pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb... [  0%] Getting: pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb... [  0%] Getting: pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb... [  0%] Getting: pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb... Downloaded 25 kiB in 25s at 1.03 kiB/s.
Everything OK. Moving meta files ...
Cleanup mirror.
All done.
Errors:
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
Failed to download files (4 errors)!

[steelhead31]

@hdurand-artic Are you using a mirror repo/proxy or anything of that nature?, also would you mind sharing your repository configurations, and specifically what/how you are downloading the jdks ? .. the read errors appear to be related to debmirror..

[hdurand-work]

@steelhead31 So, we're using debmirror to mirror every adoptium debian packages (amd64 / bullseyes-buster) - and serve them back ourselves internally.

I don't think there's a proxy between our server and your server.

(I'm personally not super familiar with debmirror)

Everything worked before the original issue happened so I really don't think it's on our end.

[steelhead31]

@hdurand-artic  I think the cache on the debmirror may need updating, as we had to re-upload some packages following an initial error... is it possible for you to try with the debmirror disabled?, so you are downloading directly.

[hdurand-work]

is it possible for you to try with the debmirror disabled?

It really isn't no, our machines don't have external access if they don't have to

[steelhead31]

I've reached out to jfrog to check everything on their end, especially the fact that the web-ui doesn't display any packages, though it does list 13 available.

[hdurand-work]

@steelhead31 So, after digging into the debmirror documentation, I found out about "precleanup" after running it once I got those errors:

Errors:
 Download of pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Failed to download files (4 errors)!

but those errors don't appear again if I run it again with or without cleanup... I guess something is still wrong on your end?

[steelhead31]

Yes, it appears there may be an issue on our hosted jfrog server, I've raised an issue with them to investigate, but it looks related to metadata/indexes on the artifactory server itself.  The packages are visible ( and installable via other methods ) . But I think I've exhausted everything I can do.. I'll update this issue when I get an update

[hdurand-work]

@steelhead31 Alright, thank you!

[pandeyv-del]

Adding more details on this.

case1: if i add repo details on the CentOS/fedora on the server then there is no error and system will able to download the package. cat <<EOF > /etc/yum.repos.d/adoptium.repo [Adoptium] name=Adoptium baseurl=https://packages.adoptium.net/artifactory/rpm/${DISTRIBUTION_NAME:-$(. /etc/os-release; echo $ID)}/\$releasever/\$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.adoptium.net/artifactory/api/gpg/key/public EOF

Cas2: if i add these in some kind of script(in my case i was doing on json for packer) and add the repo detail then first it will start throwing error around escape character echo $ID)}/\$releasever/\$basearch if we remove escape character then it will not throw any error but system will not be able to download the package saying repodata xml is not valid to resolve this i have escaped the escape character also like this baseurl=https://packages.adoptium.net/artifactory/rpm/${DISTRIBUTION_NAME:-$(. /etc/os-release; echo $ID)}/\\$releasever/\\$basearch

After this fix packer json also valid and system was able to download the package.

[steelhead31]

@pandeyv-del thanks for that information, thats useful to know...

@hdurand-artic Apologies Im still waiting on an update from Jfrog for this issue, I've chased them again this morning.  Can you have a look at the comment above and see if something similar is appropriate for your mirror scripting / environment?

[hdurand-work]

@steelhead31 Hello, do you have any news ?

By the way, here's the debmirror output with debug/verbose enabled (I had to dug a bit recently to understand how debmirror works - after an unrelated issue - so I was able to enable that)

Get Translation files ...
Get DEP-11 metadata files ...
Get command-not-found metadata files ...
Files to download: 568 MiB
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb => [  0%] Getting: pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb...     #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb ==> 200 OK (4s)
200 OK
read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb => [  0%] Getting: pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb...     #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb ==> 500 read timeout (3s)
500 read timeout
Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb => [  0%] Getting: pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb...     #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb ==> 500 read timeout (3s)
500 read timeout
Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb => [  0%] Getting: pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb...         #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb ==> 500 read timeout (3s)
500 read timeout
Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Downloaded 38 kiB in 20s at 1.93 kiB/s.
Everything OK. Moving meta files ...
Moving dists/bullseye/InRelease
Moving dists/bullseye/Release
Moving dists/bullseye/main/binary-all/Packages.gz
Moving dists/bullseye/main/binary-amd64/Packages.gz
Moving dists/bullseye/Release.gpg
Moving dists/buster/InRelease
Moving dists/buster/Release
Moving dists/buster/main/binary-all/Packages.gz
Moving dists/buster/main/binary-amd64/Packages.gz
Moving dists/buster/Release.gpg
Moving dists/bookworm/InRelease
Moving dists/bookworm/Release
Moving dists/bookworm/main/binary-all/Packages.gz
Moving dists/bookworm/main/binary-amd64/Packages.gz
Moving dists/bookworm/Release.gpg
Cleanup mirror.
deleting pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb-4017329
All done.
Errors:
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Failed to download files (4 errors)!

[hdurand-work]

So it seems to work now.

I also just noticed that URLs had double "/" "https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb" , after solving this.

It seems to be working consistently... so problem solved? Maybe? https://packages.adoptium.net/ui/packages/deb:%2F%2Ftemurin-11-jdk?name=temurin-11&type=packages still broken on JFrog though.

[steelhead31]

Thanks for the update, Im still awaiting a response from jfrog... ,so Im guessing the caching issue with broken URLs is now working properly... as far as the packages, Im pretty much stuck until jfrog take a look.. as they manage the repository, and the only option left to me, is to rebuild the repository from scratch.. which I'd like to avoid unless its absolutely necessary.

[github-actions[bot]]

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable. It will be closed soon unless the stale label is removed by a committer, or a new comment is made.

[steelhead31]

I think these issues have now been resolved. Certainly the most recent release packages work correctly.