Consistent Custom Channel `itest` Flakes

[linden]

when running make itest I'm seeing consistent flakes (roughly 9 out of 10 runs flake) on the custom_channels* tests. I've tried on both master and the v0.15.0-alpha release, on my machine and in a docker container. I'm on a fairly fast machine (2023 MacBook Pro M3 Max/128gb ram), so this is could be a race condition that the slower github actions machine isn't running into. as noted by lightninglabs/lightning-terminal#991, time.Sleep is used in quite a few places.

I've worked around these issues by adding retries to the AssetBalance checks and reverting this commit (on my fork), which passes consistently.

here's an example flake before my change to AssetBalance.

                Error Trace:    /Users/linden/go/pkg/mod/github.com/lightninglabs/[email protected]/itest/assertions.go:2323
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/assets_test.go:793
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/assets_test.go:465
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/litd_custom_channels_test.go:387
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/test_harness.go:103
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/litd_test.go:88
                Error:          Not equal: 
                                expected: 0xc3500
                                actual  : 0x0
                Test:           TestLightningTerminal/custom_channels
                Messages:       ListAssets balance, wanted 800000, got: {
                                    "assets": [],
                                    "unconfirmed_transfers": "1",
                                    "unconfirmed_mints": "0"
                                }

and after my change to AssetBalance but before reverting the commit:

    assets_test.go:1759: 
                Error Trace:    /Users/linden/Documents/apps/linden/lightning-terminal/itest/assets_test.go:1759
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/litd_custom_channels_test.go:2899
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/litd_custom_channels_test.go:2961
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/test_harness.go:103
                                                        /Users/linden/Documents/apps/linden/lightning-terminal/itest/litd_test.go:88
                Error:          Received unexpected error:
                                rpc error: code = DeadlineExceeded desc = context deadline exceeded
                Test:           TestLightningTerminal/custom_channels_liquidity

[jtobin]

This also appears to be popping up in recent LiT test failures in CI.

[ffranr]

Thanks for looking into this @linden! And for the solution you've investigated.

Can i cherry-pick your commit https://github.com/linden/taproot-assets/commit/a354337ac2c3d02a1f11d6faad4628b623e196d7 into a PR or do you mean to open a PR yourself?

[linden]

@ffranr of course! you're welcome to cherry pick them.

[ffranr]

The remaining itest flakes mentioned in this issue should all be limited to the terminal package. I will look into them when I can.

[ffranr]

litd itest flake CI fail: https://github.com/lightninglabs/lightning-terminal/actions/runs/18164359497/job/51702542960?pr=1146

[ffranr]

I’ll close this issue now that the tap-side changes have merged. Further improvements on the litd side will continue under https://github.com/lightninglabs/lightning-terminal/issues/991