Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_buster_main_binary-amd64_Packages.lz4

[nicedawg]

trafficstars

Platform: Debian Buster

Architecture: Amd64

Our docker-based CI build suddenly started failing today...

Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Ign:2 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb buster InRelease
Get:3 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb buster Release [5285 B]
Hit:4 http://deb.debian.org/debian buster InRelease
Hit:5 http://deb.debian.org/debian buster-updates InRelease
Get:6 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb buster Release.gpg [821 B]
0% [Working]Get:7 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb buster/main amd64 Packages [5058 B]
Fetched 11.2 kB in 0s (23.2 kB/s)

Reading package lists... 0%
Reading package lists... Error!

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_buster_main_binary-amd64_Packages.lz4
E: The package lists or status file could not be parsed or opened.
Exited with code 100

It sounds simliar to Issue AdoptOpenJDK/openjdk-build#1179 .

Is there something we need to do differently? Please let me know if you need more info.

Thanks!

[bailey-maitland]

Also occurring on Ubuntu 19.04:

Hit:9 https://updates.signal.org/desktop/apt xenial InRelease                                                                      
Hit:10 http://binaries.erlang-solutions.com/debian disco Release                                                         
Hit:11 https://storage.googleapis.com/bazel-apt stable InRelease                                                         
Hit:12 http://dl.google.com/linux/chrome/deb stable Release                                        
Ign:14 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb disco InRelease       
Hit:16 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb disco Release
Reading package lists... Error!
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_disco_main_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.

[nicedawg]

For anyone looking for a temporary workaround, downloading the .deb file and installing it directly (rather than using apt) seems to work well for us (but YMMV) Make sure you edit the url to get the version you need:

TEMP_DEB="$(mktemp)" && wget -O "$TEMP_DEB" 'https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/pool/main/a/adoptopenjdk-8-hotspot/adoptopenjdk-8-hotspot_8u222-b10-2_amd64.deb' && sudo dpkg -i "$TEMP_DEB"
rm -f "$TEMP_DB"```

[aahlenst]

Index has been rgenerated. Should be good, now.

[nicedawg]

I verified that it's working correctly now. Thanks so much for fixing it so quickly!

[mjp91]

Also occurring on Ubuntu 19.04:

Hit:9 https://updates.signal.org/desktop/apt xenial InRelease                                                                      
Hit:10 http://binaries.erlang-solutions.com/debian disco Release                                                         
Hit:11 https://storage.googleapis.com/bazel-apt stable InRelease                                                         
Hit:12 http://dl.google.com/linux/chrome/deb stable Release                                        
Ign:14 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb disco InRelease       
Hit:16 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb disco Release
Reading package lists... Error!
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_disco_main_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.

Still occurring in 19.04

[aahlenst]

Index has been regenerated and it's working, at least for the moment.

Still occurring in 19.04

It has been working in the meantime. But every time we upload another package it might break again. And we're uploading a lot of packages because this week the new CPUs came out. It's annoying and I'm looking for a solution.

[chabala]

I'm commenting out your PPA in the meantime. Please update the issue when it's stable again.

[bailey-maitland]

+1 for temporarily disabling it; it's affecting the Ubuntu automatic updater (resulting in visible warnings on login) and at one point was hiding a kernel update 😕

[aahlenst]

I apologize to everyone of you for the service disruption and that it is taking us so long to fix it. I'm in contact with JFrog (the creators of Artifactory that we're using to host the packages) to find a permanent solution. I'll post an update as soon as I have more information.

[Pr0methean]

The package list probably needs to be generated, and tested by reading it into apt, on AdoptOpenJDK's fantastic CI/CD system.

[aahlenst]

TL;DR: It's safe to use our package feed again.

The problem seems to be the incremental metadata generation in Artifactory. Together with JFrog, we've been able to reliably reproduce and isolate the issue. But there's no fix yet. Until there's a permanent fix, I've disabled the automatic upload of releases. We'll generate and upload those manually for the time being.

I hope we can sort these issues out until the next CPU.

[aahlenst]

Looks like this issue is going to stay with us a little while longer. JFrog plans to develop and release a fix "in the following two quarters".

[Pr0methean]

This makes it sound to me like maybe JFrog isn't the best choice of provider.

[planetf1]

Interestingly we have a weird issue with jfrog on our project too. I know they're looking into it but I'll be rather concerned if it takes them 2Q to address ( https://github.com/odpi/egeria/issues/1659 )

[tgamauf]

It seems like this issue popped up again for Ubuntu 18.04

[aahlenst]

@tgamauf Yes, deliberately this time. I pushed a round of releases. Takes 1 hour to 90 minutes per major JDK version. It completed a few minutes ago, so the indexes have been rebuilt already.

[tgamauf]

Right, works like a charm now. So I was really just unlucky, I guess ;) Thanks for your quick response!

[Pr0methean]

@aahlenst Shouldn't it be possible to point users at the old version until the new one is built and uploaded (e.g. by using a URL rewrite rule to track what version is current), so that the updates don't affect uptime at all?

[aahlenst]

@Pr0methean We use a hosted instance of Artifactory. We cannot change the configuration of the webserver. Problem should be solved in the not so distant future: https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1002

[aahlenst]

The problem was apparently fixed (wasn't notified). No manual metadata generation was required during the July release cycle. Therefore I close this ticket.

We'll use something else for Adoptium that's more reliable.

[Sternenforscher]

I can confirm, same message as @overlisted but with

Encountered a section with no Package: header
Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_buster_main_binary-amd64_Packages
Die Paketliste oder die Statusdatei konnte nicht eingelesen oder geöffnet werden.
Can't call method "policy" on an undefined value at /usr/bin/apt-show-versions line 53.
Traceback (most recent call last):
  File "/usr/bin/debtags", line 700, in <module>
    main()
  File "/usr/bin/debtags", line 694, in main
    sys.exit(action.main(args))
  File "/usr/bin/debtags", line 632, in main
    for pkg, tags in self.tags_from_apt():
  File "/usr/bin/debtags", line 256, in tags_from_apt
    cache = self.apt_cache
  File "/usr/bin/debtags", line 241, in apt_cache
    res = self._apt_cache = apt.Cache()
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 170, in __init__
    self.open(progress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 230, in open
    self._cache = apt_pkg.Cache(progress)
apt_pkg.Error: E:Encountered a section with no Package: header, E:Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_buster_main_binary-amd64_Packages, E:The package lists or status file could not be parsed or opened.
Paketlisten werden gelesen... Fehler!
E: Problem executing scripts APT::Update::Post-Invoke-Success 'test -x /usr/bin/apt-show-versions || exit 0 ; apt-show-versions -i'
E: Sub-process returned an error code
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_buster_main_binary-amd64_Packages
E: Die Paketliste oder die Statusdatei konnte nicht eingelesen oder geöffnet werden.

the last 2 lines are what overlisted posted but in german.

[withanhdammit]

And for me:

Hit:8 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb buster InRelease
Reading package lists... Error!
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/adoptopenjdk.jfrog.io_adoptopenjdk_deb_dists_buster_main_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.

[aahlenst]

https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1647