build-bootstrap.sh: fix extract_debs

[segfault-bilibili]

https://github.com/termux/termux-packages/issues/12529#issuecomment-1288599164

in the current broken state of build-bootstrap.sh, the -f flag is necessary to build a correct bootstrap zip.

Maybe this is a fix.

[sylirre]

Maybe this is a fix.

This is not a fix. The build script doesn't check for deb files. Instead it checks for the "mark" files in /data/data/.built-packages. The -f option disabled this check and lets to build a package even if it exists.

https://github.com/termux/termux-packages/blob/78edcc80551995aa028c6a07f6b91c9d6e5b4246/scripts/build/termux_step_start_build.sh#L52-L63

As you can see only package name and version are checked. Not the architecture.

Personally I would be against making separate directory for each architecture as default. It is not necessary and can be easily implemented manually through the -o option if workflow requires doing so. Otherwise it is quite easy to filter deb files by architecture part in their file name (pkgname_version_architecture.deb, deb file name format as described by Debian guidelines).

[sylirre]

The directory .built-packages as well as whole prefix is being moved into ~/.termux-build/_databackups when build-package.sh switches architecture through option -a. Script build-bootstraps.sh doesn't handle that currently. Proper fix would be looking into that instead of distributing deb files by multiple directories.

[segfault-bilibili]

The build script doesn't check for deb files. Instead it checks for the "mark" files in /data/data/.built-packages. As you can see only package name and version are checked. Not the architecture.

I see /data/data/.built-packages being symlinked to corresponding "current" arch, so that I think maybe I can separate deb files in the same way.

Otherwise it is quite easy to filter deb files by architecture part in their file name

Currently I don't see such filtering in extract_debs():

https://github.com/termux/termux-packages/blob/78edcc80551995aa028c6a07f6b91c9d6e5b4246/scripts/build-bootstraps.sh#L115-L130

Personally I would be against making separate directory for each architecture as default.

Then such filtering may be added in extract_debs()?

However, personally I think that would seem neat - a simple * wildcard covers all deb files required for an arch. Then it seems easy for me to clear built deb files according to -f in build-bootstrap.sh, as well as other possible scenarios like keeping an archive of all deb required for each arch (which could be used for manual install etc).

[segfault-bilibili]

Changing this variable will also break our automated builds in GitHub Actions.

I once changed /output to a symlink, like: /output => /output-multiarch/aarch64 - however I think that would require an extra line in .gitignore. Maybe it would be fine to just symlink /output to somewhere else.

[segfault-bilibili]

The directory .built-packages as well as whole prefix is being moved into ~/.termux-build/_databackups when build-package.sh switches architecture through option -a.

Sorry, but I don't get what you mean.

To my understanding, extracting deb files is possibly a kind of clean install.

Proper fix would be looking into that instead of distributing deb files by multiple directories.

Would that be the same as extracting deb files?

[sylirre]

Summary of needed changes:

1. Loop over deb files with necessary architecture only. So if we need bootstrap for arm, the loop should run over *_${current_arch}.deb and *_all.deb instead of *.deb. This will let to filter only deb files relevant to selected architecture.
2. Handle built packages in exactly same way as done by build-package.sh.

Implementing both doesn't require splitting deb files on architecture-specific directories.

---

Previously build-bootstraps.sh was a standalone script. Now it is getting more and more integrated into package build system and it isn't necessary to do operations differently than being done by build-package.sh.

[sylirre]

Replaced by https://github.com/termux/termux-packages/pull/17031

[segfault-bilibili]

Um, I'm sorry. I didn't understand this issue correctly. I once thought this to be like: deb files of different arch being erroneously mixed together - it's not that simple.

Today I got into a "strange" situation that .built-packages "mysteriously" being modified to point to an incorrect arch which contained marker files I had already deleted. Then it turned out to be exactly the reason (which I once failed to understand) you had outlined:

The directory .built-packages as well as whole prefix is being moved into ~/.termux-build/_databackups when build-package.sh switches architecture through option -a.

But then I think there could be a disconnection between .built-packages and output.

I wonder whether deb file names have exact one-to-one mapping to marker files currently placed in built-packages?

If so, I think the deb file itself can already represent the status that whether a package has already been built.

[segfault-bilibili]

In termux_step_start_build.sh, there's a hint being printed to the terminal which says: the marker file could be manually deleted in order to force a rebuild: https://github.com/termux/termux-packages/blob/763d9d499312eadfea0b51b14adbe573919e91c9/scripts/build/termux_step_start_build.sh#L55

However in build-bootstrap.sh, the symlink always gets deleted on exit: https://github.com/termux/termux-packages/blob/763d9d499312eadfea0b51b14adbe573919e91c9/scripts/build-bootstraps.sh#L241-L243

Then it turns out that things behind are much more complicated...

[sylirre]

The reason why build-bootstraps works incorrectly is because it implements OWN method to deal with built packages. Each architecture must have:

• Own set of marker files.
• Own TERMUX_PREFIX.

All of these should be switched automatically according of the selected architecture. So need to use termux_step_handle_buildarch function instead of manually trying to change the arch. In other words same as done by build-package.sh script.

This pull request is closed because it is wrong.

Please check this instead: https://github.com/termux/termux-packages/pull/17031. It addresses the architecture issues.

[segfault-bilibili]

Ah there's package manager other than apt, there's on-device-build... I need to look into this deeper.

[segfault-bilibili]

This pull request is closed because it is wrong.

Sorry again, but I'm already aware.

I'm just thinking of if it's possible to simply get rid of marker files completely, or in other words, use deb file names as markers directly.

[AhmTaher]

https://https://github.com/termux/termux-packages/blob/4c8f9453779203b100a3918746482f2cadbf4d44/scripts%2Fbuild-bootstraps.sh

[sylirre]

@segfault-bilibili I think this can be done

[segfault-bilibili]

I've deleted my lengthy comments, because I don't want to occupy your time to just explain things to me.