Loop out fee sent, HTLCs remains pending yet LITD says it failed

[MButcho]

trafficstars

I looped out 5 mil sats with custom BTC address and no confirmation target. After 7 hours, HTLCs are still pending, fee was paid 7 hours ago but https://terminal.lightning.engineering/liquidity/ says it failed.

Actually loop out the amount to custom BTC address, I have done it many times before

1. Choose 5 050 000 sats with slider
2. Click Loop (one time manual)
3. Click advanced, entered custom bc1q... BTC address, leave confirmation target empty
4. Clicked finish or confirm or something like that
5. Retarted LITD

Happend for the 1st time.

Standalone

lnd-mode=remote

LiT version: 0.12.2-alpha commit=v0.12.2-alpha / website

physical HW, SSD

Ubuntu 22.04 LTS

Loop Out Amount 5,050,000 sats Status Failed Failure Reason Off-chain Failure Initiated Jan 25, 2024 1:37 PM Last Update Jan 25, 2024 2:56 PM ID 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a HTLC Address bc1plmdpe96jdk4xh922f4w2gfngq3y3kpgk3ezx3nk2vsgs3xp9ew3q65d8se Swap Server Node Pubkey 021c97a90a411ff2b10dc2a8e32de2f29d2fa49d41bfbb52bd416e460db0747d0d

Fee Payment to: 03fb2a0ca79c005f493f (7 hours ago) 30,000 Created: 7 hours ago (2024.01.25 - 13:37:46) Destination Node: 03fb2a0ca79c005f493f Fee: 48 Fee msats: 48000 millisats Hops: 2 Hop 1: cyberdyne.sh Hop 2: LOOP Id: d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db Is Outgoing: true Secret: xxx M Tokens: 30000000 millisats

2024-01-25 13:37:43.673 [INF] LOOPD: Loop out request received
2024-01-25 13:37:43.701 [INF] LOOP: LoopOut 0.05050000 BTC to bc1q... (channels: )
2024-01-25 13:37:43.968 [INF] LOOP: Initiating swap request at height 827310: amt=0.05050000 BTC, expiry=827410
2024-01-25 13:37:44.369 [INF] LOOP: 896ec1 Htlc address (P2TR): bc1plmdpe96jdk4xh922f4w2gfngq3y3kpgk3ezx3nk2vsgs3xp9ew3q65d8se
2024-01-25 13:37:44.415 [INF] LOOP: 896ec1 Server message: Use AutoLoop - it is really good, give it a try
2024-01-25 13:37:44.418 [INF] LOOP: 896ec1 Loop out swap state: Initiated
2024-01-25 13:37:44.421 [INF] LOOP: 896ec1 Sending swap payment lnbc50263260n1pjmy4scpp539hvzkm32v9jqe75307k8pjmcxupej960ljqqqk95vuxscu2mx9qdy9wdmkzupq95s8xcmjd9c8gw3qx5cnyvrxv4jxzvtr8ymn2v3kv3skzdnz8y6ngcf5vs6kxcf5xgmrvwpsxs6rjvtzxq6nzd3cv56rgd3cvdjkxcfkxscnzvpc8yurydtrvfsnycqzxgxq97zvuqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37smapyqqqqqqqq2qqqqqqqqqqqqqqq9qsp5sn2wq73qrcr3q3h7v9eqr49dllzmmaa6xmc24996jssxwrwf7ksq9qyyssqx3u9kk8jswkn44c8nzxw03heez9tcfr4kzf8mz62qn66l42z48lp7wjvdkd9d75ldyc5qe09u9vwx0cyltdgsqw3mkm02fhc7wlfgvgpx2naal
2024-01-25 13:37:44.642 [INF] LOOP: 896ec1 Server update: SERVER_INITIATED received, timestamp: 2024-01-25 13:37:44.245419666 +0100 CET
2024-01-25 13:37:44.649 [INF] LOOP: 896ec1 Server recommended routing plugin: Low/High
2024-01-25 13:37:44.649 [INF] LOOP: 896ec1 Sending prepayment lnbc300u1pjmy4scpp56tmpca0dkvhtsg8e7qaygvjug2dytextylle3689qal0rw07zldsdq2wpex2urp0ycqzxgxq97zvuqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37smapyqqqqqqqq2qqqqqqqqqqqqqqq9qsp5d6243690c6t73hdxv9ffsahvxfk5mu88hn99zzx4k28cjlt0s4hq9qyyssqnx4k50g7tdrfz4gan5f0dgz7z2q8d8lwl38z0sv0ftnndtqzxaq99dwphk23lsxa60qeed8jppty07chul6fydntyexv5slcpr3790sp908jxg
2024-01-25 13:37:44.650 [INF] LOOP: 896ec1 Register 1 conf ntfn for swap script on chain (hh=827310)
2024-01-25 13:37:44.653 [INF] LOOP: 896ec1 Checking preimage reveal height 827390 exceeded (height 827310)
2024-01-25 13:37:44.653 [INF] LOOP: 896ec1 Waiting for either htlc on-chain confirmation or off-chain payment failure
2024-01-25 13:37:46.459 [INF] LOOP: 896ec1 Acquired routing plugin Low/High for payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a
2024-01-25 13:37:46.460 [INF] LOOP: 896ec1 Payment (896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a) try count 1/3 (plugin=Low/High)
2024-01-25 13:37:46.460 [INF] LOOP: 896ec1 Payment (d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db) try count 1/1 (plugin=None)
2024-01-25 13:37:46.500 [INF] LOOP: 896ec1 Payment d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db: state=IN_FLIGHT, inflight_htlcs=0, inflight_amt=0 mSAT
2024-01-25 13:37:46.500 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=0, inflight_amt=0 mSAT
2024-01-25 13:37:46.764 [INF] LOOP: 896ec1 Payment d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db: state=IN_FLIGHT, inflight_htlcs=1, inflight_amt=30000000 mSAT
2024-01-25 13:37:46.799 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=1, inflight_amt=2513163000 mSAT
2024-01-25 13:37:46.856 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=2, inflight_amt=3769744500 mSAT
2024-01-25 13:37:46.915 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=3, inflight_amt=5026326000 mSAT
2024-01-25 13:37:47.291 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=2, inflight_amt=3769744500 mSAT
2024-01-25 13:37:47.352 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=3, inflight_amt=5026326000 mSAT
2024-01-25 13:37:47.922 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=2, inflight_amt=3769744500 mSAT
2024-01-25 13:37:47.991 [INF] LOOP: 896ec1 Payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a: state=IN_FLIGHT, inflight_htlcs=3, inflight_amt=5026326000 mSAT
2024-01-25 13:37:48.967 [INF] LOOP: 896ec1 Server update: SERVER_HTLC_PUBLISHED received, timestamp: 2024-01-25 13:37:48.862091221 +0100 CET
2024-01-25 13:38:14.647 [ERR] LOOP: 896ec1 swap subscription error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

After reboot

2024-01-25 14:56:04.974 [INF] RPCS: Starting RPC Server
2024-01-25 14:56:04.974 [INF] LITD: Starting LiT macaroon service
2024-01-25 14:56:05.111 [INF] AUTO: Starting Autopilot Client
2024-01-25 14:56:05.394 [INF] LOOP: 896ec1 Acquired routing plugin Low/High for payment 896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a
2024-01-25 14:56:05.395 [INF] LOOP: 896ec1 Payment (d2f61c75edb32eb820f9f03a44325c429a45e4cb27ff98e8e5077ef1b9fe17db) try count 1/1 (plugin=None)
2024-01-25 14:56:05.395 [INF] LOOP: 896ec1 Payment (896ec15b71530b2067d48bfd63865bc1b81cc8ba7fe40002c5a33868638ad98a) try count 1/3 (plugin=Low/High)
2024-01-25 14:56:05.410 [INF] LOOP: 896ec1 Failed prepayment: rpc error: code = Unknown desc = payment is in transition
2024-01-25 14:56:05.410 [INF] LOOP: 896ec1 Canceling swap: 1 payment failed: FAILURE_REASON_NONE, 0 attempts
2024-01-25 14:56:05.860 [INF] LOOP: 896ec1 Wait for server pulling off-chain payment(s)
2024-01-25 14:56:05.860 [INF] LOOP: 896ec1 Swap payment failed: rpc error: code = Unknown desc = payment is in transition
2024-01-25 14:56:05.860 [INF] LOOP: 896ec1 Swap completed: FailOffchainPayments (final cost: server 0 BTC, onchain 0 BTC, offchain 0 BTC)
2024-01-25 14:56:05.867 [INF] LOOP: 896ec1 Loop out swap state: FailOffchainPayments

EDIT: highlighted the logs