mutiny-web icon indicating copy to clipboard operation
mutiny-web copied to clipboard

Error: failed to read data from storage

Open Mahue93 opened this issue 1 year ago • 23 comments

I’m getting this error after downloading the mutiny iOS app. Mutiny wallet isn’t open on any browsers. I’d appreciate any feedback or suggestion on how to troubleshoot, thanks. mutiny-logs.txt

Mahue93 avatar Mar 09 '24 18:03 Mahue93

I’m getting this error after downloading the mutiny iOS app. Mutiny wallet isn’t open on any browsers. I’d appreciate any feedback or suggestion on how to troubleshoot, thanks.

mutiny-logs.txt

Web logs

496e 2024-03-10 03:03:41.996 TRACE [mutiny_core::nodemanager:314] Checking device lock 496e 2024-03-10 03:03:41.996 INFO [mutiny_core::nodemanager:316] Current device lock: DeviceLock { time: 1710039817, device: "5868a4d0-3d69-4d88-884f-7bc1dcb6e066" } 496e 2024-03-10 03:03:42.230 TRACE [mutiny_core::nodemanager:319] Device lock set: took 234ms 496e 2024-03-10 03:03:42.230 INFO [mutiny_core::nodemanager:342] Building node manager components 496e 2024-03-10 03:03:42.278 DEBUG [mutiny_core::gossip:181] Previous gossip sync timestamp: 0 496e 2024-03-10 03:03:42.279 TRACE [mutiny_core::gossip:221] Gossip sync/Scorer initialized in 1ms 496e 2024-03-10 03:03:42.279 TRACE [mutiny_core::nodemanager:385] Node manager Components built: took 49ms 496e 2024-03-10 03:03:42.279 DEBUG [mutiny_core::nodemanager:404] Building nodes 496e 2024-03-10 03:03:42.279 INFO [mutiny_core::node:341] initializing a new node: Some("49bb4da6-aa26-46aa-a4b1-f24d976ebbde") 496e 2024-03-10 03:03:42.303 INFO [mutiny_core::node:401] creating lsp client 496e 2024-03-10 03:03:42.303 INFO [mutiny_core::node:409] lsp config matches saved lsp config 496e 2024-03-10 03:03:42.303 DEBUG [mutiny_core::ldkstorage:206] Reading channel manager from storage 496e 2024-03-10 03:03:42.305 INFO [lightning::ln::channelmanager:10363] Successfully loaded channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e at update_id 15 against monitor at update id 14 496e 2024-03-10 03:03:42.305 ERROR [lightning::ln::channelmanager:10643] A ChannelMonitor is stale compared to the current ChannelManager! This indicates a potentially-critical violation of the chain::Watch API! 496e 2024-03-10 03:03:42.305 ERROR [lightning::ln::channelmanager:10644] The ChannelMonitor for channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e is at update_id 14 with update_id through 14 in-flight 496e 2024-03-10 03:03:42.305 ERROR [lightning::ln::channelmanager:10646] but the ChannelManager is at update_id 15. 496e 2024-03-10 03:03:42.305 ERROR [lightning::ln::channelmanager:10647] The chain::Watch API requires that monitors are persisted durably before returning, 496e 2024-03-10 03:03:42.305 ERROR [lightning::ln::channelmanager:10648] client applications must ensure that ChannelMonitor data is always available and the latest to avoid funds loss! 496e 2024-03-10 03:03:42.305 ERROR [lightning::ln::channelmanager:10649] Without the latest ChannelMonitor we cannot continue without risking funds. 496e 2024-03-10 03:03:42.305 ERROR [lightning::ln::channelmanager:10650] Please ensure the chain::Watch API requirements are met and file a bug report at https://github.com/lightningdevkit/rust-lightning 3d7a 2024-03-10 13:16:06.524 TRACE [mutiny_core::nodemanager:314] Checking device lock 3d7a 2024-03-10 13:16:06.524 INFO [mutiny_core::nodemanager:316] Current device lock: DeviceLock { time: 1710076564, device: "5868a4d0-3d69-4d88-884f-7bc1dcb6e066" } 3d7a 2024-03-10 13:16:06.743 TRACE [mutiny_core::nodemanager:319] Device lock set: took 219ms 3d7a 2024-03-10 13:16:06.743 INFO [mutiny_core::nodemanager:342] Building node manager components 3d7a 2024-03-10 13:16:06.784 DEBUG [mutiny_core::gossip:181] Previous gossip sync timestamp: 0 3d7a 2024-03-10 13:16:06.785 TRACE [mutiny_core::gossip:221] Gossip sync/Scorer initialized in 1ms 3d7a 2024-03-10 13:16:06.785 TRACE [mutiny_core::nodemanager:385] Node manager Components built: took 42ms 3d7a 2024-03-10 13:16:06.785 DEBUG [mutiny_core::nodemanager:404] Building nodes 3d7a 2024-03-10 13:16:06.785 INFO [mutiny_core::node:341] initializing a new node: Some("49bb4da6-aa26-46aa-a4b1-f24d976ebbde") 3d7a 2024-03-10 13:16:06.809 INFO [mutiny_core::node:401] creating lsp client 3d7a 2024-03-10 13:16:06.809 INFO [mutiny_core::node:409] lsp config matches saved lsp config 3d7a 2024-03-10 13:16:06.809 DEBUG [mutiny_core::ldkstorage:206] Reading channel manager from storage 3d7a 2024-03-10 13:16:06.811 INFO [lightning::ln::channelmanager:10363] Successfully loaded channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e at update_id 15 against monitor at update id 14 3d7a 2024-03-10 13:16:06.811 ERROR [lightning::ln::channelmanager:10643] A ChannelMonitor is stale compared to the current ChannelManager! This indicates a potentially-critical violation of the chain::Watch API! 3d7a 2024-03-10 13:16:06.811 ERROR [lightning::ln::channelmanager:10644] The ChannelMonitor for channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e is at update_id 14 with update_id through 14 in-flight 3d7a 2024-03-10 13:16:06.811 ERROR [lightning::ln::channelmanager:10646] but the ChannelManager is at update_id 15. 3d7a 2024-03-10 13:16:06.811 ERROR [lightning::ln::channelmanager:10647] The chain::Watch API requires that monitors are persisted durably before returning, 3d7a 2024-03-10 13:16:06.811 ERROR [lightning::ln::channelmanager:10648] client applications must ensure that ChannelMonitor data is always available and the latest to avoid funds loss! 3d7a 2024-03-10 13:16:06.811 ERROR [lightning::ln::channelmanager:10649] Without the latest ChannelMonitor we cannot continue without risking funds. 3d7a 2024-03-10 13:16:06.811 ERROR [lightning::ln::channelmanager:10650] Please ensure the chain::Watch API requirements are met and file a bug report at https://github.com/lightningdevkit/rust-lightning 5079 2024-03-10 21:54:16.963 TRACE [mutiny_core::nodemanager:314] Checking device lock 5079 2024-03-10 21:54:16.963 INFO [mutiny_core::nodemanager:316] Current device lock: DeviceLock { time: 1710107654, device: "5868a4d0-3d69-4d88-884f-7bc1dcb6e066" } 5079 2024-03-10 21:54:17.284 TRACE [mutiny_core::nodemanager:319] Device lock set: took 321ms 5079 2024-03-10 21:54:17.284 INFO [mutiny_core::nodemanager:342] Building node manager components 5079 2024-03-10 21:54:17.330 DEBUG [mutiny_core::gossip:181] Previous gossip sync timestamp: 0 5079 2024-03-10 21:54:17.331 TRACE [mutiny_core::gossip:221] Gossip sync/Scorer initialized in 1ms 5079 2024-03-10 21:54:17.331 TRACE [mutiny_core::nodemanager:385] Node manager Components built: took 47ms 5079 2024-03-10 21:54:17.331 DEBUG [mutiny_core::nodemanager:404] Building nodes 5079 2024-03-10 21:54:17.331 INFO [mutiny_core::node:341] initializing a new node: Some("49bb4da6-aa26-46aa-a4b1-f24d976ebbde") 5079 2024-03-10 21:54:17.356 INFO [mutiny_core::node:401] creating lsp client 5079 2024-03-10 21:54:17.356 INFO [mutiny_core::node:409] lsp config matches saved lsp config 5079 2024-03-10 21:54:17.356 DEBUG [mutiny_core::ldkstorage:206] Reading channel manager from storage 5079 2024-03-10 21:54:17.357 INFO [lightning::ln::channelmanager:10363] Successfully loaded channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e at update_id 15 against monitor at update id 14 5079 2024-03-10 21:54:17.357 ERROR [lightning::ln::channelmanager:10643] A ChannelMonitor is stale compared to the current ChannelManager! This indicates a potentially-critical violation of the chain::Watch API! 5079 2024-03-10 21:54:17.358 ERROR [lightning::ln::channelmanager:10644] The ChannelMonitor for channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e is at update_id 14 with update_id through 14 in-flight 5079 2024-03-10 21:54:17.358 ERROR [lightning::ln::channelmanager:10646] but the ChannelManager is at update_id 15. 5079 2024-03-10 21:54:17.358 ERROR [lightning::ln::channelmanager:10647] The chain::Watch API requires that monitors are persisted durably before returning, 5079 2024-03-10 21:54:17.358 ERROR [lightning::ln::channelmanager:10648] client applications must ensure that ChannelMonitor data is always available and the latest to avoid funds loss! 5079 2024-03-10 21:54:17.358 ERROR [lightning::ln::channelmanager:10649] Without the latest ChannelMonitor we cannot continue without risking funds. 5079 2024-03-10 21:54:17.358 ERROR [lightning::ln::channelmanager:10650] Please ensure the chain::Watch API requirements are met and file a bug report at https://github.com/lightningdevkit/rust-lightning 21f8 2024-03-11 15:19:41.213 TRACE [mutiny_core::nodemanager:314] Checking device lock 21f8 2024-03-11 15:19:41.213 INFO [mutiny_core::nodemanager:316] Current device lock: DeviceLock { time: 1710170378, device: "5868a4d0-3d69-4d88-884f-7bc1dcb6e066" } 21f8 2024-03-11 15:19:41.434 TRACE [mutiny_core::nodemanager:319] Device lock set: took 221ms 21f8 2024-03-11 15:19:41.434 INFO [mutiny_core::nodemanager:342] Building node manager components 21f8 2024-03-11 15:19:41.480 DEBUG [mutiny_core::gossip:181] Previous gossip sync timestamp: 0 21f8 2024-03-11 15:19:41.481 TRACE [mutiny_core::gossip:221] Gossip sync/Scorer initialized in 1ms 21f8 2024-03-11 15:19:41.481 TRACE [mutiny_core::nodemanager:385] Node manager Components built: took 47ms 21f8 2024-03-11 15:19:41.481 DEBUG [mutiny_core::nodemanager:404] Building nodes 21f8 2024-03-11 15:19:41.482 INFO [mutiny_core::node:341] initializing a new node: Some("49bb4da6-aa26-46aa-a4b1-f24d976ebbde") 21f8 2024-03-11 15:19:41.506 INFO [mutiny_core::node:401] creating lsp client 21f8 2024-03-11 15:19:41.506 INFO [mutiny_core::node:409] lsp config matches saved lsp config 21f8 2024-03-11 15:19:41.506 DEBUG [mutiny_core::ldkstorage:206] Reading channel manager from storage 21f8 2024-03-11 15:19:41.508 INFO [lightning::ln::channelmanager:10363] Successfully loaded channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e at update_id 15 against monitor at update id 14 21f8 2024-03-11 15:19:41.508 ERROR [lightning::ln::channelmanager:10643] A ChannelMonitor is stale compared to the current ChannelManager! This indicates a potentially-critical violation of the chain::Watch API! 21f8 2024-03-11 15:19:41.508 ERROR [lightning::ln::channelmanager:10644] The ChannelMonitor for channel d1d9e350bfeda15c5f5ec02ff53dfa3fd1b78ea76dc9d98c30a37678c5b90f8e is at update_id 14 with update_id through 14 in-flight 21f8 2024-03-11 15:19:41.508 ERROR [lightning::ln::channelmanager:10646] but the ChannelManager is at update_id 15. 21f8 2024-03-11 15:19:41.508 ERROR [lightning::ln::channelmanager:10647] The chain::Watch API requires that monitors are persisted durably before returning, 21f8 2024-03-11 15:19:41.508 ERROR [lightning::ln::channelmanager:10648] client applications must ensure that ChannelMonitor data is always available and the latest to avoid funds loss! 21f8 2024-03-11 15:19:41.508 ERROR [lightning::ln::channelmanager:10649] Without the latest ChannelMonitor we cannot continue without risking funds. 21f8 2024-03-11 15:19:41.508 ERROR [lightning::ln::channelmanager:10650] Please ensure the chain::Watch API requirements are met and file a bug report at https://github.com/lightningdevkit/rust-lightning

Mahue93 avatar Mar 11 '24 15:03 Mahue93

Do you have any older logs from this wallet? It seems you've hit on a critical bug in either LDK or Mutiny, and we'd definitely like to dig further into this.

TheBlueMatt avatar Mar 11 '24 17:03 TheBlueMatt

It looks like the last time the channels updated were on 2024-02-29. Do you still have the instance of the wallet that worked back then? You said you're also (or originally) using it on the web? Was it working there as well at one time but not working anymore? Is it not working there anymore because you also restored from seed words on the web too?

AnthonyRonning avatar Mar 11 '24 17:03 AnthonyRonning

Unfortunately this is going to require a custom solution on our side to fix and release and we aren't planning on a release for some time. If you have the original wallet where it was working, that's about all that can be done at the moment. This has yet to happen to another user and we're not sure how this happened. Any details you can provide and logs from potentially other devices/browsers you might have used the wallet from would help. Sorry for the inconvenience.

AnthonyRonning avatar Mar 11 '24 18:03 AnthonyRonning

Thank you all for looking into this. As soon as I'm able I will check and see if I have any earlier logs and see if there are any other details I can provide.

Mahue93 avatar Mar 11 '24 21:03 Mahue93

Unfortunately I don't have any older logs than the ones I already submitted. I also don't have any working version either on web or the iOS app. Everything was working fine and I recently closed a small initial channel to make a slightly larger channel. Before restoring to the new iOS app, I closed out my mobile web browser wallet. That's when I received the error. I can be patient until a new release is out with a potential bug fix. Though if it ends up being a long time, is there a way to close/force close my channel with the lsp and then just sweep the on chain funds? Thanks.

Mahue93 avatar Mar 12 '24 12:03 Mahue93

I closed out my mobile web browser wallet

I presume you just mean you closed the tab? I'm assuming by its not working here you mean you tried to reopen it and it fails similarly to the iOS app?

TheBlueMatt avatar Mar 12 '24 14:03 TheBlueMatt

I closed out my mobile web browser wallet

I presume you just mean you closed the tab? I'm assuming by its not working here you mean you tried to reopen it and it fails similarly to the iOS app?

Yes, I closed the tab Exactly, after receiving the error on the iOS app, I deleted the app and tried restoring on the web browser but received the same error.

Mahue93 avatar Mar 12 '24 18:03 Mahue93

Unfortunately this is going to require a custom solution on our side to fix and release and we aren't planning on a release for some time. If you have the original wallet where it was working, that's about all that can be done at the moment. This has yet to happen to another user and we're not sure how this happened. Any details you can provide and logs from potentially other devices/browsers you might have used the wallet from would help. Sorry for the inconvenience.

Just to confirm, there's not really anything for me to do at this point besides wait for a future release is that correct?

Mahue93 avatar Mar 15 '24 13:03 Mahue93

Unfortunately this is going to require a custom solution on our side to fix and release and we aren't planning on a release for some time. If you have the original wallet where it was working, that's about all that can be done at the moment. This has yet to happen to another user and we're not sure how this happened. Any details you can provide and logs from potentially other devices/browsers you might have used the wallet from would help. Sorry for the inconvenience.

Is there any update or timeline on a potential fix? I'm still continuing to see the same issue. Thanks.

Mahue93 avatar Apr 09 '24 13:04 Mahue93

Thanks for the reminder. We haven't done anything here yet since it's fairly advanced and custom to do. How much was the amount you have stuck on there?

AnthonyRonning avatar Apr 09 '24 14:04 AnthonyRonning

Thanks for the reminder. We haven't done anything here yet since it's fairly advanced and custom to do. How much was the amount you have stuck on there?

Of course, and I understand there's a lot going on with Mutiny right now. Approximately 1.5 mil sats give or take.

Mahue93 avatar Apr 09 '24 15:04 Mahue93

I'm having a similar message. I am attaching the logs I have. It seems to be showing "ChannelMonitor for channel is missing" Uploading mutiny-logs.txt…

tinygiant avatar Apr 11 '24 02:04 tinygiant

@tinygiant something went wrong with uploading your logs, but if you could open a separate issue that'd be preferred

benthecarman avatar Apr 11 '24 02:04 benthecarman

Thanks for the reminder. We haven't done anything here yet since it's fairly advanced and custom to do. How much was the amount you have stuck on there?

Of course, and I understand there's a lot going on with Mutiny right now. Approximately 1.5 mil sats give or take.

Would you be comfortable running a custom command line tool to recover your funds for this?

AnthonyRonning avatar Apr 12 '24 05:04 AnthonyRonning

I've very rarely used the CLI but I can give it a shot. Is this going to require closing the channel to fix?

Mahue93 avatar Apr 12 '24 15:04 Mahue93

Yes, that's a requirement either way.

AnthonyRonning avatar Apr 12 '24 15:04 AnthonyRonning

Yes, that's a requirement either way.

Sounds good, I'm ready to try. Thanks again.

Mahue93 avatar Apr 12 '24 17:04 Mahue93

Yes, that's a requirement either way.

Hello, just checking to see if there is any update or new info on this. Thanks.

Mahue93 avatar Apr 25 '24 21:04 Mahue93

I also figured out more details about the funding tx including date, size etc. if that helps getting the channel closed.

Mahue93 avatar Apr 29 '24 16:04 Mahue93

Efforts are being made right now to make mutiny capable of running as a server / command line tool. So unfortunately that's the big blocker here. It's actively being worked on though, and afterwards we can give you instructions of how to fix this situation. Has only ever happened to one other person and it's hard to build straight into the wallet for this edge case.

AnthonyRonning avatar Apr 29 '24 16:04 AnthonyRonning

Efforts are being made right now to make mutiny capable of running as a server / command line tool. So unfortunately that's the big blocker here. It's actively being worked on though, and afterwards we can give you instructions of how to fix this situation. Has only ever happened to one other person and it's hard to build straight into the wallet for this edge case.

I understand, thanks for giving me an update!

Mahue93 avatar Apr 29 '24 16:04 Mahue93

Yeah, In my case (referred to by benthecarman above, "Issues--> "Delete Everything"), neither restoration by seed words nor "import state from file" was successful in recovering wallet. Seed words result in "failed to read data from storage," and saved state-file results in "load timed out, please try again."

A little over 3 million sats sent to Mutiny wallet on-chain from hardware wallet, swapped to lightning BTC inside Mutiny wallet using swap button for that entry in self-custody funds list, lightning channel then created automatically in Mutiny, but with 100% outbound and 0% inbound liquidity. No recovery or access so far.

stevednm avatar May 07 '24 16:05 stevednm

Hello, I'm just checking back to see if there has been any progress or a fix I may have missed. Thanks.

Mahue93 avatar Jun 08 '24 13:06 Mahue93

Hey, I got our recovery tool working again, you can check it out here: https://github.com/MutinyWallet/mutiny-data-recovery/releases/tag/v0.1.1

This will force close your channels and allow you to recover your funds on-chain

benthecarman avatar Jun 08 '24 15:06 benthecarman

Hey, I got our recovery tool working again, you can check it out here: https://github.com/MutinyWallet/mutiny-data-recovery/releases/tag/v0.1.1

This will force close your channels and allow you to recover your funds on-chain

Thank you!

Mahue93 avatar Jun 08 '24 16:06 Mahue93

This worked for me. :+1: Was able to successfully launch Mutiny wallet (PWA version in Firefox, Linux) after using recovery tool, which force closed the lightning channel, freeing the funds locked in that channel that I could not access. Will check Mutiny every few days to see when I can transfer the BTC back to an on-chain wallet.

Great fix for cases where Mutiny wallet cannot load for some reason and wallet funds become inaccessible. THANK YOU!

stevednm avatar Jun 08 '24 17:06 stevednm

Awesome glad it was able to fix!

benthecarman avatar Jun 08 '24 17:06 benthecarman

Hey, I got our recovery tool working again, you can check it out here: https://github.com/MutinyWallet/mutiny-data-recovery/releases/tag/v0.1.1

This will force close your channels and allow you to recover your funds on-chain

Does anyone have a quick tutorial on how to execute this for someone not familiar with Mac terminal? In the meantime I'll try to figure it out.

Mahue93 avatar Jun 08 '24 17:06 Mahue93

Download the darwin version, then open a terminal in the folder that you download it to and type ./mutiny-data-recovery_darwin

benthecarman avatar Jun 08 '24 17:06 benthecarman