XPUBS param no longer accepts "rescan" option in BWT 0.2.0

[Stadicus]

Adding XPUBs and rescanning the Bitcoin blockchain is the one big drawback of lightweight Electrum middleware like BWT and EPS. This should be as easy as possible for the user, or even 3rd-party software to add XPUBs automatically.

The bwt.env config file seems like a good place to manage the keys that are monitored. I was happy to see that the following works in BWT 0.1.5:

• adding multiple XPUBS= lines, not needing to concatenate the very long xpub strings into one, which would be a maintenance nightmare
• adding the rescan arguemnt to the xpub, like on the command line: XPUBS='zpub12345....XYZ:2020-10-01'

While the first point still seems to work with BWT 0.2-RC1, the second usage gives now an error:

error: Invalid value for '--xpub <xpubs>...': invalid base58 character 0x3a

I think the option to easily add a new XPUB and specify the "rescan since" option just for that XPUB is very important. Adding it directly to the xpub string seems the easiest way to manage the keys.

Is this an unintended change? Or, if now, would it be feasible to make that work again?

[Stadicus]

Seems like I was mistaken. BWT accepts multiple XPUBS lines in the config without error, but seems to scan only the first one.

I'd love to see the option to have one xpub per line, with the rescan date, in the bwt.env config file:

• much easier to add new xpubs over time with copy/paste
• better to maintain and keep an overview
• more feasible for 3rd-party-apps to add keys automatically

[shesek]

Seems like I was mistaken. BWT accepts multiple XPUBS lines in the config without error, but seems to scan only the first one.

That is correct. The config file uses the dotenv format, which is a simple flat list of key-value environment variables that can't conveniently represent options with multiple values.

I can definitely see how this is not ideal though... the reason I implemented it like that was the the cli arguments parser library I'm using (structopt) does not support any form of config files, but does support configuration via environment variables, so it was easiest to inject the options via that.

I could quite easily support JSON config files (or YAML, at the cost of introducing another dependency) that are used in place of the structopt cli options, such that you'll have to configure everything in the JSON file. Do you think this would be useful?

Supporting using both a config file and cli options together would either require replacing structopt or getting a config file PR merged to it (or adding a bunch of hairy code to bwt that uses structopt in a way it was not intended to -- at which point its probably better to just replace it).

adding the rescan arguemnt to the xpub, like on the command line: XPUBS='zpub12345....XYZ:2020-10-01'

This is something that was changed recently, because Bitcoin Core only really cares about the earliest timestamp within the importmulti batch, and to simplify the CLI/FFI interfaces. But I'm open to changing that back, is there a reason that you think I should now that you know the rational behind it?

Also note that you really only have to specify --rescan-since when adding new xpubs. When starting up with xpubs that are all already imported, it won't issue any rescans so the rescan timestamp doesn't matter.

[shesek]

Looking at the documentation of the library I'm using to load the dotenv file, it looks like its possible to do something like this:

XPUBS=xpub1234
XPUBS=$XPUBS;ypub5678
XPUBS=$XPUBS;zpub9999

(Note that the separator was changed from , to ;. This is because of the newly added support for descriptors, which can contain ,s in them. I added a mention of this to the CHANGELOG.)

[shesek]

Another possible hack is to create a file with a list of cli arguments. This might be nicer because it also supports short options and doesn't SCREAM AT YOU. For example:

-n regtest
--xpub xpub1234
-x ypub5678
--descriptor wpkh(xpub4321/0/*)
-d wpkh(zpub9999/0/*) -d wpkh(zpub9999/1/*)

Then start with bwt `cat /path/to/args` . This could also be combined with additional options, i.e. bwt `cat /path/to/args` --poll-interval 2 ...

[Stadicus]

Maybe I need to take a step back and quickly explain why I think BWT is an awesome small program from a regular user's perspective:

• Running a dedicated node (e.g. on a Raspberry Pi) is not for everyone
• The option to just have Bitcoin Core on your machine and connect any wallet that supports that to it using BWT is really helpful to gain privacy with hardware wallets
• BWT is great in that regard because it's a simple binary almost everyone can run (as opposed to EPS that needs a Python runtime environment)
• As a regular user, I'd like to have an installer, a config directory with a simple file to edit and a start menu entry and as little exposure to the command line as possible (but that's of course about five steps ahead... :grin:)

In regard of the configuration file, both options work already really well:

• using XPUBS is a bit screamy, but simple enough. If it's just the bwt.env file (maybe call it bwt.conf or something more self-explanatory?) that I need to edit, and I can add XPUBs over time, that would simplyfy xpub key management already a lot.

• the file with CLI arguments works equally well, although the format is a bit uncommon for config files. The one thing I think should be avoided is the need to call it with other cli arguments like cat and the like. That's not very noob-friendly. Maybe there could be an option like --config /path/to/bwt.conf?

The reason why I would like to see the "rescan since" option directly with the xpub is that it's fire and forget. If you add a new wallet, you add one line. BWT could safely ignore it after that (even if it maybe implies that changing the rescan date on an existing xpub would trigger a rescan). Having the RESCAN_SINCE argument is a bit strange, because it's not really a configuration parameter, but more a one-time command. And that would also trigger a rescan only for newly added xpubs, IIUC.

If I could take all your options from above and define the config file format I personally would like the best, it would probably be this:

• ~/bwt.conf

  XPUBS=xpub1234
  XPUBS=xpub5678:2020-11-01
  XPUBS=ypub4444:2020-10-01;zpub5555:2020-10-01
  

• or ./bwt --config ~/bwt.conf

  --xpub xpub1234
  --xpub xpub5678:2020-11-01
  --xpub ypub4444:2020-10-01;zpub5555:2020-10-01
  

IIUC, the first suggestion wouldn't work, but needs XPUBS=$XPUBS;... which again is more shell-script than config file. In that case, I'd probably opt for the section option.

Thanks for being open to my suggestions. I know they are not really coming from the current possibilities, but maybe it helps to get an outside user's perspective.

BWT is a great tool already in any case, and thanks so much for that!