Test many bitcoind instances

[RCasatta]

Investigation for https://github.com/lightningdevkit/rust-lightning/issues/2438

I can reproduce locally a similar error as mentioned in the issue with a big amount of processes like in the included test. It seems there is a Transport failure in the create_wallet, consequently causing the failure of load_wallet. The included commit passes the added tests but doesn't fix it, it just considers ok if the default wallet is not created. In another branch I tried to repeat the create_wallet but the error seems unrecoverable.

Not sure how to properly fix, maybe adding a Config flag to avoid the creation of the default wallet, delegating it to clients is an option

[tnull]

Mh, looking at this again since we're still regularly see failing tests like:

thread 'test_esplora_syncs' panicked at 'called `Result::unwrap()` on an `Err` value: JsonRpc(Rpc(RpcError { code: -4, message: "Wallet file verification failed. SQLiteDatabase: Unable to obtain an exclusive lock on the database, is it being used by another instance of Bitcoin Core?\n", data: None }))', tests/integration_tests.rs:27:70

Isn't the fundamental issue here that it tries to access the locked SQLite database, likely a leftover from a (failed) prior instance? Possibly this could be fixed by making the cleanup code more robust?

[RCasatta]

Any idea to make the cleanup code more robust?

tempfile crate should take care of creating a unique data dir and clean the dir once the ref goes of out scope

[tnull]

Any idea to make the cleanup code more robust?

tempfile crate should take care of creating a unique data dir and clean the dir once the ref goes of out scope

Yes, but it sometimes doesn't seem to do its job properly? Naively I'd wonder if there is maybe another way we can detect whether the encoutered directory is stale and just delete it before startup if it is?