termux-packages icon indicating copy to clipboard operation
termux-packages copied to clipboard

enhance(scripts/generate-bootstraps.sh): add caching support

Open truboxl opened this issue 3 years ago • 4 comments

Since build-bootstraps.sh is out of commission and pending a rewrite (https://github.com/termux/termux-packages/issues/10462#issuecomment-1205603274), I decided to just improve generate-bootstraps.sh which already does not include extra packages when generating bootstraps to begin with.

With the optional caching turned on, it gained a tiny bit of speed up. Time lost most of which is spend on downloading the same *_all.deb files again thrice. But subsequent runs will show massive speed up since files are stored and cached.

Without caching all arch (default)
real	20m57.026s
user	0m0.515s
sys	0m0.870s
With caching all arch (1st run)
real	19m14.773s
user	0m0.508s
sys	0m0.972s
With caching all arch (after 1st run)
real	0m54.656s
user	0m0.084s
sys	0m0.100s

With caching also allows building for custom arches. For my arm vfpv3-d16 project, I can imagine triggering multiple parallel CI running ./scripts/run-docker.sh ./build-package.sh -a arm one-bootstrap-package-per-CI and then pass all deb files to ./scripts/run-docker.sh ./scripts/generate-bootstraps.sh -c --architectures arm for processing and not rely on build-bootstraps.sh anymore...

Note that file lists are not cached.

truboxl avatar Aug 06 '22 08:08 truboxl

Will merge this soon if no objection Caching is disabled by default. The caching feature is actively being tested over at my repo https://github.com/truboxl/termux-packages-altarch

If we do enable caching in generate bootstrap ci, we should be able to cut down fetching the same *_all.deb 3 times. From my testing, they are:

Package: termux-am
Download-Size: 17.1 kB
Package: ca-certificates
Download-Size: 234 kB
Package: termux-keyring
Download-Size: 39.2 kB
Package: termux-licenses
Download-Size: 52.2 kB
Package: termux-tools
Download-Size: 27.9 kB

which should save 371.4kB x 3 = 1111.2kB bandwidth if enabled, probably too small to notice Might want to explore more methods to cut down bandwidth to the server in the future

truboxl avatar Oct 24 '22 09:10 truboxl

After months of commits, are you sure you are done? ;)

It's a good idea to cache for people who have space, so thanks.

But there are changes required.

agnostic-apollo avatar Oct 24 '22 12:10 agnostic-apollo

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Dec 09 '22 05:12 stale[bot]