[bug]: sync issue with grouped asset

[jharveyb]

Background

When trying to sync all issuance proofs from the default Lightning Labs testnet universe, my sync hangs and then fails.

Your environment

tapcli getinfo                       # version of `tapd`, `lnd`, and network

tapd v0.3.1, lnd v0.17.1, testnet

uname -mrsv                          # operating system 

debian testing, 16-core CPU, NVMe SSD

bitcoind --version || btcd --version # version of `btcd`, `bitcoind`, or other backend

local bitcoind with neutrino support + other neutrino peers

• any other relevant environment details

Steps to reproduce

Sync all issuance proofs from LL testnet default universe.

Relevant log lines:

Last universe root mentioned before the hang (could be a different root since sync is parallel):

2023-11-15 13:29:32.378 [DBG] UNIV: UniverseRoot(issuance-6a35df8d68de999b0b4a878f91e4074eb2410bca9a9fabaf0ad0c46ae0627d8d): Inserting 200 new leaves (200 of 4410)

Log lines that cover the sync hang and failure:

2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.473 [TRC] PROF: Derived Taproot Asset commitment taproot_asset_root=86fdcd1a1ff35f8474e3e6e2995ebfcfd9f0842ec621c07641265f93922438df, internal_key=0349f518ab7d8e124482c806d3d2c5316d1b7d3dd7adea01c3e60eb49f3df4a403, taproot_key=f1372944ab1a37c5b61ad40f14d178f21021d3270a8dc881608d1c131a62a62a
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.472 [TRC] CMMT: Derived asset inclusion proof for asset_id=b165c1c272e14599d992662f63d90a8b445d77650faf29f4db9440a090fb5f7b, asset_commitment_key=cd548bb2a6660bd241f8cfc34d9542b31658eec44bc4c2525a45b9e925eca6cf, asset_commitment_leaf=0e540facfbd553a8c990e89e6a761aff72f3715f079aa76581fed5ea2ffb1d33
2023-11-15 13:29:32.473 [TRC] PROF: Derived Taproot Asset commitment taproot_asset_root=86fdcd1a1ff35f8474e3e6e2995ebfcfd9f0842ec621c07641265f93922438df, internal_key=0349f518ab7d8e124482c806d3d2c5316d1b7d3dd7adea01c3e60eb49f3df4a403, taproot_key=f1372944ab1a37c5b61ad40f14d178f21021d3270a8dc881608d1c131a62a62a
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.472 [TRC] PROF: Derived Taproot Asset commitment taproot_asset_root=86fdcd1a1ff35f8474e3e6e2995ebfcfd9f0842ec621c07641265f93922438df, internal_key=0349f518ab7d8e124482c806d3d2c5316d1b7d3dd7adea01c3e60eb49f3df4a403, taproot_key=f1372944ab1a37c5b61ad40f14d178f21021d3270a8dc881608d1c131a62a62a
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.472 [TRC] PROF: Derived Taproot Asset commitment taproot_asset_root=86fdcd1a1ff35f8474e3e6e2995ebfcfd9f0842ec621c07641265f93922438df, internal_key=0349f518ab7d8e124482c806d3d2c5316d1b7d3dd7adea01c3e60eb49f3df4a403, taproot_key=f1372944ab1a37c5b61ad40f14d178f21021d3270a8dc881608d1c131a62a62a
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.472 [TRC] PROF: Derived Taproot Asset commitment taproot_asset_root=86fdcd1a1ff35f8474e3e6e2995ebfcfd9f0842ec621c07641265f93922438df, internal_key=0349f518ab7d8e124482c806d3d2c5316d1b7d3dd7adea01c3e60eb49f3df4a403, taproot_key=f1372944ab1a37c5b61ad40f14d178f21021d3270a8dc881608d1c131a62a62a
2023-11-15 13:29:32.473 [TRC] PROF: Derived Taproot Asset commitment taproot_asset_root=86fdcd1a1ff35f8474e3e6e2995ebfcfd9f0842ec621c07641265f93922438df, internal_key=0349f518ab7d8e124482c806d3d2c5316d1b7d3dd7adea01c3e60eb49f3df4a403, taproot_key=f1372944ab1a37c5b61ad40f14d178f21021d3270a8dc881608d1c131a62a62a
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.473 [TRC] PROF: Derived Taproot Asset commitment taproot_asset_root=86fdcd1a1ff35f8474e3e6e2995ebfcfd9f0842ec621c07641265f93922438df, internal_key=0349f518ab7d8e124482c806d3d2c5316d1b7d3dd7adea01c3e60eb49f3df4a403, taproot_key=f1372944ab1a37c5b61ad40f14d178f21021d3270a8dc881608d1c131a62a62a
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:29:32.473 [TRC] PROF: Verifying tapscript proof
2023-11-15 13:30:14.539 [ERR] RPCS: [/universerpc.Universe/SyncUniverse]: unable to sync universe: unable to register proofs: unable to register proofs: unable to verify issuance proofs: unable to verify proof: group key not known

Group anchors should always be verified and inserted before other group members, and that should happen within the same goroutine, so I'm not sure how we're reaching this situation where proof verification is incorrectly ordered.