Multi-architecture image builds failing

[WadeBarnes]

As a result of the fix in https://github.com/hyperledger/aries-cloudagent-python/pull/2123, for https://github.com/hyperledger/aries-cloudagent-python/issues/2121, the extras, [askar,bbs], are now being installed and this is causing the multi-architecture image builds to fail, specifically the ones for linux/arm64 and linux/386.

The failures arise due to the fact that the visions of the askar and bbs packages defined in the requirements files do not have wheel packages published for the linux/arm64 and linux/386 architectures and they do not have source packages published that would allow for the packages to be built and installed from source. In the case of the askar dependencies, compatible wheel packages have been published for other versions, just not the ones defined currently.

Steps to reproduce locally:

• Get the latest aca-py code.

• Run either of the following commands

  docker buildx build --no-cache --progress plain -t main --platform linux/386  -f docker/Dockerfile -t aca-py-test-build .
  docker buildx build --no-cache --progress plain -t main --platform linux/arm64  -f docker/Dockerfile -t aca-py-test-build .
  

• Review the build failure messages.

• In contrast building the linux/amd64 image using the following command succeeds without error:

  docker buildx build --no-cache --progress plain -t main --platform linux/amd64  -f docker/Dockerfile -t aca-py-test-build .
  

Example Results:

The error messages change depending on the order pip tries to resolve the bbs and askar dependencies.

docker buildx build --no-cache --progress plain -t main --platform linux/arm64 -f docker/Dockerfile -t aca-py-test-build .

...
#17 [main  9/10] RUN aries_cloudagent_package=$(find ./ -name "aries_cloudagent*.whl" | head -n 1) &&     echo "Installing ${aries_cloudagent_package} ..." &&     pip install --no-cache-dir --find-links=. ${aries_cloudagent_package}[askar,bbs] &&     rm aries_cloudagent*.whl
#17 0.516 Installing ./aries_cloudagent-0.8.0rc0-py3-none-any.whl ...
#17 13.60 Looking in links: .
#17 13.66 Processing ./aries_cloudagent-0.8.0rc0-py3-none-any.whl
...
#17 58.73 ERROR: Could not find a version that satisfies the requirement ursa-bbs-signatures~=1.0.1; extra == "bbs" (from aries-cloudagent[askar,bbs]) (from versions: none)
#17 58.73 ERROR: No matching distribution found for ursa-bbs-signatures~=1.0.1; extra == "bbs"
...

docker buildx build --no-cache --progress plain -t main --platform linux/386 -f docker/Dockerfile -t aca-py-test-build .

...
#17 [main  9/10] RUN aries_cloudagent_package=$(find ./ -name "aries_cloudagent*.whl" | head -n 1) &&     echo "Installing ${aries_cloudagent_package} ..." &&     pip install --no-cache-dir --find-links=. ${aries_cloudagent_package}[askar,bbs] &&     rm aries_cloudagent*.whl
#17 0.477 Installing ./aries_cloudagent-0.8.0rc0-py3-none-any.whl ...
#17 1.884 Looking in links: .
...
#17 19.30 ERROR: Could not find a version that satisfies the requirement indy-credx~=0.3; extra == "askar" (from aries-cloudagent[askar,bbs]) (from versions: 0.2.1, 0.2.3)
#17 19.30 ERROR: No matching distribution found for indy-credx~=0.3; extra == "askar"...

[WadeBarnes]

cc @andrewwhitehead, @dbluhm, @swcurran

[WadeBarnes]

The obvious fix is to update the publishing for the affected askar and bbs requirements and update the requirements.bbs.txt, and requirements.askar.txt files. However I'm not familiar with the work or effort needed.

[WadeBarnes]

For the short term I'm going to submit a PR to the associated workflows so they only produce the linux/amd64 layer so we can publish some working images for testing.

[swcurran]

I believe the solution will be to bump the versions — the new builds for the libraries I think support multiple platforms. We need to get the versions that support that officially released. Is that correct @andrewwhitehead ?

[WadeBarnes]

ursa-bbs-signatures (https://github.com/mattrglobal/ffi-bbs-signatures) does not publish supporting packages.

[swcurran]

That’s not good. Can we remove BBS+ from ACA-Py unless explicitly added? E.g. separate it out the way indy-idk and Askar are included?

[WadeBarnes]

I should qualify my last statement. They publish packages that are compatible with the linux/amd64 builds, but none to support the linux/arm64 and linux/386 builds.

[WadeBarnes]

Summary of work that needs to be done to support multiarchitecture aca-py images:

• The published packages for aries-askar, indy-credx, and indy-vdr need to be updated/released with support for linux/arm64 and linux/386; either in binary or source form.

• The requirements.askar.txt file needs to be updated to reference the updated releases.

• In order to support BBS+, the same steps need to be taken with the published packages for ursa-bbs-signatures (requirements.bbs.txt) which are associated with the mattrglobal/ffi-bbs-signatures project.

• Finally support for multiarchitecture builds must be reenabled in the publish workflows, as it was disabled by https://github.com/hyperledger/aries-cloudagent-python/pull/2125.

[swcurran]

To be a bit more precise, given the current state of where we want to go -- indy-credx needs to be replaced by anoncreds-rs, and the version of anoncreds-rs used needs to have support for the different architectures.

Sigh...

[vitorestevamia]

Any news on it? The multi-arch images are still disabled?

[WadeBarnes]

Yes, they are still disabled. We're waiting on the multi-arch support to be added to the dependencies as listed above.

[esune]

Is this something we could tackle next, based on the anoncreds-rs work and having non-indy images now? @swcurran @WadeBarnes @andrewwhitehead

[swcurran]

The issue is with the BBS Signatures dependency. The work needed to do that is (at a high level) documented in this issue — https://github.com/hyperledger/aries-bbssignatures-rs/issues/8. I was going to see if @amanji could do that one prior to his next big assignment.

[esune]

The issue is with the BBS Signatures dependency. The work needed to do that is (at a high level) documented in this issue — hyperledger/aries-bbssignatures-rs#8. I was going to see if @amanji could do that one prior to his next big assignment.

Thanks for the clarification. Yes, this would be a good "filler" if capacity allows for it, I will let @amanji decide on that.

[amanji]

As we're still waiting on items for the next assignment, I've spoken with Andrew and will make a move on this today.

[biagioboi]

Just for completeness. I have tried to install aires-cloudagent-python on Raspberry Pi 4 (ARM Cortex-A53 processor) by manually building libraries for aries-askar indy-credx and indy-vdr. For indy-vdr and aries-askar I do not faced any problem. For indy-credx the compilation is successful

But when I try to release new credentials or a new scheme I receive an error from the bindings.py wrapper script

I have tried with indy-credx==1.0.0, please let me know if a workaround exists for solving this problem or if you are working on it .. or if it can be a problem related to the version used (I do not tink so because it only appears on raspberry, while docker works perfectly!)

[esune]

Adding that this seems to also (obviously) affect devcontainers when running on an arm64 platform such as MacOS - they fail to build and start with ursa-bbs-signatures being the trigger of the error.

[swcurran]

That’s not good — the dependency on ursa-bbs-signatures should have been removed and replaced with aries-bbs-signatures. That change won't solve the arm64 issue, but I thought that change had been made a year ago or so.

@andrewwhitehead — can you please list the steps needed to move the BBS Signatures such that it would be up to date and support arm64 platforms? What work should be done to get there?

Thanks!

[andrewwhitehead]

aries-bbssignatures-rs actually just replaces the bbs crate which ursa-bbs-signatures (aka ffi-bbs-signatures) has as a dependency. There were a few changes made in the process, but it looks like switching the wrapper crate over to use aries-bbssignatures should be straightforward (it's compiling locally for me).

The other steps are listed in the linked issue but aries-bbssignatures needs to be published, first, and then the release workflow for ffs-bbs-signatures updated to publish arm64 packages.

[swcurran]

From the ACA-Pug calls, here are the range of options for solving this by removing BBS Signatures from the primary path.

• Create Docker images that don't have BBS SIgnatures Signatures included, and provide guidance on install time changes needed to add it.
  • Kind of like a plugin – deployers must explicitly add BBS vs. expecting to be there.
  • Variation: Multiple tags – e.g. one with a different tag for "with BBS" and "without BBS".
    • What goes in the current tag, what goes in the version with the new tag? No BBS in current, BBS in the new tag.
• Switch to another implementation?
  • Multiple, non-interoperable implementations. It's not clear which one to switch to.
• THE "CORRECT" OPTION – BUT MORE WORK Move current implementation to a plugin
  • Allows those using it (is there anyone?) to use the plugin.
  • This is the pattern we are moving to – removing from core.
  • Allows for other implementations in the same pattern.
• Drop the support for BBS entirely for now.
  • The "next gen" BBS signatures implementations are coming – wait for them and add back then.

Comments from chat:

• 08:50:41 From Daniel Bluhm : I have the same question as Patrick; if it's not getting used or is pretty far out of spec, is it worth going through the effort of keeping it going?
• 08:50:51 From Colton Wolkins : #4 drops it completely, instead of allowing it as an option (from my understanding)
• 08:52:45 From Daniel Bluhm : Plugin-ifying would probably require the most development effort to execute
• 08:53:07 From Daniel Bluhm : There are some hacky spots to support BBS keys that would need to be addressed
• 08:56:43 From Daniel Bluhm : We've been able to work around it for our development focus for those on my team using M*, though it is a frequent headache still. Just not blocking.
• 08:57:53 From Colton Wolkins : Isn't it by running the containers in amd64 mode or something like that?
• 08:57:54 From Emiliano Suñé : "We've been able to work around it for our development focus for those on my team using M*, though it is a frequent headache still. Just not blocking". Do you have tips/recommendations we may want to include in the docs in the meantime for this? I usually happen to switch to using arm64 right after I forgot what I did the time before to get it working...
• 08:58:46 From Emiliano Suñé : If you are running the image you can specify the platform, but if you are developing (i.e.: devcontainer) it will complain about the architecture when installing dependencies in the first place since the proper binaries are not available
• 09:00:07 From Colton Wolkins : I don't have a Mac, so I can't 100% vouch for this, but I'm seeing this export internally @Emiliano Suñé : export DOCKER_DEFAULT_PLATFORM=linux/amd64
• 09:00:08 From Colton Wolkins : [This is an encrypted message]
• 09:00:14 From Daniel Bluhm : We haven't gotten into the habit of using devcontainers (yet) at least but yeah, it's that platform flag that we use, specifying linux/amd64

[rblaine95]

Just to add 2 cents to this.

I tested building this on my M2 Mac and can confirm 1.0.0rc6 builds successfully 🎉 All previous versions do not build on my local.

docker build -t acapy-agent -f ./docker/Dockerfile .

[rblaine95]

I've tested re-enabling the linux/arm64 platform in CI

https://github.com/didx-xyz/aries-cloudagent-python/actions/runs/10419734030/job/28858389568

Building and pushing the regular image works, but the extended image seems to fail.

#36 19.80 INFO: pip is looking at multiple versions of aries-cloudagent[askar,bbs,didcommv2] to determine which version is compatible with other requirements. This could take a while.
#36 19.82 ERROR: Could not find a version that satisfies the requirement ursa-bbs-signatures<1.1.0,>=1.0.1; extra == "bbs" (from aries-cloudagent[askar,bbs,didcommv2]) (from versions: none)
#36 20.45 ERROR: No matching distribution found for ursa-bbs-signatures<1.1.0,>=1.0.1; extra == "bbs"

linux/386 still fails

[swcurran]

The extended image will not work for multi-architecture, because it includes BBS (as your log indicates), and it is the lack of a multi-architecture BBS artifact that prevents multi-architecture from working. So I think what you are seeing is expected.