🐛[Bug] - When running in multi-ledger configuration, a read-only agent will fail to start unless it's "main" ledger has a TAA

[esune]

When running ACA-Py in read-only mode with a multi-ledger configuration, the "root/main" ledger needs to have TAA enabled otherwise the agent will fail to start with the following error:

aca-py-1         | 2024-03-07 18:12:30,041 aries_cloudagent.config.ledger INFO Fetching genesis transactions from: https://raw.githubusercontent.com/ICCS-ISAC/dtrust-reconu/main/CANdy/test/pool_transactions_genesis
aca-py-1         | 2024-03-07 18:12:30,297 aries_cloudagent.core.profile INFO Create profile manager: askar
aca-py-1         | 2024-03-07 18:12:31,919 aries_cloudagent.commands.start ERROR Exception during startup:
aca-py-1         | Traceback (most recent call last):
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/ledger/indy_vdr.py", line 372, in _submit
aca-py-1         |     request_result = await self.pool.handle.submit_request(request)
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/indy_vdr/pool.py", line 116, in submit_request
aca-py-1         |     result = await fut
aca-py-1         |   File "/usr/local/lib/python3.9/asyncio/futures.py", line 284, in __await__
aca-py-1         |     yield self  # This tells Task to wait for completion.
aca-py-1         |   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
aca-py-1         |     future.result()
aca-py-1         |   File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result
aca-py-1         |     raise self._exception
aca-py-1         | indy_vdr.error.VdrError: Request failed: client request invalid: could not authenticate, verkey for UD8zmp7yAphbu8rLaFfmuw cannot be found
aca-py-1         |
aca-py-1         | The above exception was the direct cause of the following exception:
aca-py-1         |
aca-py-1         | Traceback (most recent call last):
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/config/ledger.py", line 155, in ledger_config
aca-py-1         |     await wallet.set_did_endpoint(public_did, endpoint, ledger)
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/wallet/askar.py", line 458, in set_did_endpoint
aca-py-1         |     attrib_def = await ledger.update_endpoint_for_did(
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/ledger/indy_vdr.py", line 734, in update_endpoint_for_did     
aca-py-1         |     await self._submit(attrib_req, True, True)
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/ledger/indy_vdr.py", line 374, in _submit
aca-py-1         |
aca-py-1         | Shutting down
aca-py-1         |     raise LedgerTransactionError("Ledger request error") from err
aca-py-1         | aries_cloudagent.ledger.error.LedgerTransactionError: Ledger request error
aca-py-1         |
aca-py-1         | The above exception was the direct cause of the following exception:
aca-py-1         |
aca-py-1         | Traceback (most recent call last):
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/commands/start.py", line 72, in init
aca-py-1         |     await startup
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/commands/start.py", line 28, in start_app
aca-py-1         |     await conductor.setup()
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 178, in setup
aca-py-1         |     if not await ledger_config(
aca-py-1         |   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/config/ledger.py", line 157, in ledger_config
aca-py-1         |     raise ConfigError(x_ledger.message) from x_ledger  # e.g., read-only
aca-py-1         | aries_cloudagent.config.base.ConfigError: Ledger request error
aca-py-1 exited with code 0

The agent starts successfully when starting in single ledger configuration targeting the same ledger that does not have a TAA enabled.

Edit: updated to reflect the investigation performed by @WadeBarnes below. Original issue has been moved to https://github.com/hyperledger/aries-cloudagent-python/issues/2829

[WadeBarnes]

For context: this error surfaced when turning off the TAA on CANdy/BCovrin ledgers and attempting to run VC-AuthN (which uses ACA-Py in read-only mode) with a ledger configuration specifying one of those ledgers as is_write. Using a ledger such as Sovrin testnet with TAA enabled as write ledger allows the agent to start-up normally.

To confirm, the TAAs on CANdy-Test and CANdy-Prod were disabled yesterday. No other ledgers were affected. The BCovrin ledgers have never had a TAA or AML registered.

[WadeBarnes]

Which version of ACA-Py was being used? Did you try the same procedure with a previous version of ACA-Py for comparison?

[WadeBarnes]

No issues were found when testing with ACA-Py v0.10.3 agents direct connected to (not using ledgers.yml) BCovrin-Test, CANdy-Test and Sovrin TestNet. ACAPY_READ_ONLY_LEDGER=false in all cases.

• askar secure storage

[WadeBarnes]

No issues were found when testing with ACA-Py v0.7.4 with an agent using a ledgers.yml file listing BCovrin-Test, CANdy-Test and Sovrin TestNet, and using Sovrin TestNet as the production (write) ledger (ACAPY_READ_ONLY_LEDGER=false).

• indy secure storage

[WadeBarnes]

No issues were found when testing with ACA-Py v0.7.4 with an agent using a ledgers.yml file listing BCovrin-Test, CANdy-Test and Sovrin TestNet, and using CANd-Test as the production (write) ledger (ACAPY_READ_ONLY_LEDGER=false).

• askar secure storage

[WadeBarnes]

No issues were found when testing with ACA-Py v0.7.4 with an agent using a ledgers.yml file listing BCovrin-Test, CANdy-Test and Sovrin TestNet, and using CANd-Test as the production (write) ledger (ACAPY_READ_ONLY_LEDGER=false).

• askar secure storage

No issues were found with the same agent when switching to ACAPY_READ_ONLY_LEDGER=true

[WadeBarnes]

No issues were found when testing with ACA-Py v0.7.4 with an agent using a ledgers.yml file listing BCovrin-Test, CANdy-Test and Sovrin TestNet, and using CANd-Test as the production (write) ledger (ACAPY_READ_ONLY_LEDGER=false).

• askar secure storage

No issues were found with the same agent when switching to ACAPY_READ_ONLY_LEDGER=true

No issues were found with a similar agent when starting it with ACAPY_READ_ONLY_LEDGER=true the first time.

• indy secure storage.

[WadeBarnes]

Odd, I tried starting up vc-authn locally with ACA-Py v0.10.3 and got the reported error. However the issue does not seem to affect existing agent deployments.

[WadeBarnes]

Same issue with ACA-Py v0.7.4. Issues occur with askar and indy storage.

[WadeBarnes]

Simply switching the is_write: true ledger to be either CANdy-Dev or BCovrin-Test, both of which have never had a TAA registered, as no affect. The error still occurs.

[WadeBarnes]

Leaving the is_write: true ledger set to BCovrin-Test and removing CANdy-Test from the list has no affect either. So this does not appear to be an issue with the having disabled the TAA on CANdy-Test.

[WadeBarnes]

Removing CANdy-Test and Sovrin TestNet from the list, the only two ledgers that have/had TAAs registered, has no affect either.

[esune]

I am not sure why this issue started happening when it did since we did not change anything in the application code that is failing, however it seems to be related to setting a wallet seed without specifying that the DID is going to be local by using --wallet-local-did. When setting a seed AND indicating the did will be local everything works as expected.

I will close this issue and update the other project(s) to follow this pattern.

@WadeBarnes we might have to review our services as this has been our go-to pattern for as long as I can remember.