LibreELEC.tv icon indicating copy to clipboard operation
LibreELEC.tv copied to clipboard

Replace tools/download-cleaner

Open antonlacon opened this issue 1 year ago • 2 comments

This replaces tools/download-cleaner with tools/download-cleaner.py. Their intended purpose is the same, but the previous bash script had limitations. The main one is that it was cumbersome in considering multiple project/device/arches.

The replacement may consider all project/device/arch combinations (see builds_all), a limited portion of those with build directories (build.LibreELEC-RPi2.arm...), or the single build specified by PROJECT=X DEVICE=Y ARCH=Z ./download-cleaner.py. Each build combination takes around 30s to consider what packages are needed, so running --all by itself is around 30 minutes.

The way it works it by:

  1. create a build plan for each build combination chosen
  2. run pkginfo against each package specified in the build plan for the desired download filename
  3. compare tarballs found in the sources directory against what the buildplan wants (ex: dav1d-1.2.0.tar.xz is present, but tree is on dav1d-1.3.0.tar.xz, then add dav1d-1.2.0.tar.xz to list of unneeded files)
  4. delete file if asked (--delete)

By default, it only prints out what it would remove.

scripts/genbuildplan.py is extended with two additional CLI options to assist with this: --hide-header prevents printing the statistics header in the output --list-packages has it output only a list of packages - no depends or build steps

Work in progress because it doesn't consider packages removed from the tree yet. I'll add a commit to drop tools/download-cleaner when this replacement is ready. Probably still useful to someone else in the meantime, so sharing.

antonlacon avatar Nov 03 '23 05:11 antonlacon

This should be ready to replace tools/download-cleaner. The two features I didn't re-implement were downloading the current source tarball for an outdated package, and building a new sources directory.

Fetching outside of a build is a repeat of tools/download-tool (though it looks like it could use some attention), or this can be extended at a later date to replace that too.

Building a new sources dir was intended as a scriptable workaround for wanting source tarballs to multiple builds. Should no longer be needed.

The full list of builds (see L42) could use a second look. I thought there were more aarch64 builds.

antonlacon avatar Nov 06 '23 22:11 antonlacon

Added --export and --import to accommodate using between different git branches. --export will save the list of package names + package source tarballs to a file. --import will take one or more of the exported files and add them to its list of files to keep.

Switched more of builds_all over to aarch64.

antonlacon avatar Dec 23 '23 21:12 antonlacon