[bug]: 0.15.3 requires additional sats to spend a UTXO completely when P2TR

[alexbosworth]

Background

I have a method that tries to find the maximum that can be spent in X UTXOs for the purpose of spending the entire UTXO.

In LND 0.15.2 and before, (including on P2TR outputs), my estimation method could produce a funded PSBT that spent down completely a UTXO, but on LND 0.15.3 my estimation method fails and requires additional funds

Your environment

• LND 0.15.3

Steps to reproduce

1. Create a template PSBT spend by selecting N inputs to spend and then send 546 (DUST) sats to a P2TR address (FundPsbt)
2. This creates a funded PSBT, now unlock the inputs
3. Sum all of the outputs of the funded PSBT to see how much it is sending out
4. Sum the outputs that are not change and then subtract to see what the total change value is
5. Fund a new PSBT with the same spend to P2TR but the output instead of 546 put as the amount (546 + total change)

Expected behavior

In LND 0.15.2 and before, the change value represents what could have been added to the main output. Funding the PSBT works by adding in the change over into the main value and then re-funding.

Actual behavior

In LND 0.15.3 in the case of sending to P2TR, trying to spend the change value as 1 output results in an error saying could not add change address to database: fee estimation not successful: insufficient funds available to construct transaction

In reality I am trying to remove the need for a change address by figuring out what can be spent into a single output given a set of UTXOs, but this is already a known issue.

Workaround

This error does not happen in 0.15.2 or when sending to P2WPKH outputs, but in the case of P2TR on LND 0.15.3 the fix is to take the fee rate multiplied by 10 and subtract that from the final output amount, any more sats spent than that will result in the error.

[guggero]

I tried to reproduce this on regtest and wasn't able to. Did I follow your instructions correctly?

Here's the first PSBT I created, with a 546 sat output sent to a P2TR address:

cHNidP8BAPgCAAAABKL8nNtV6rdlliID0zj3d0il+gkr5B8BWk4u0ST8lFMuAAAAAAAAAAAA39rFflQzhHIZLyEiybvHV4PcPeT4jIkKSIxjeR89PloAAAAAAAAAAADouZgo7pEglYdiqOKjRH7JVc1mXWRyT4YgGoMxwlXD5QAAAAAAAAAAAAF34Eb/J9G6akhqZ163e3aH/U8u+bk4NG+QJOkbnqfmAAAAAAAAAAAAAiICAAAAAAAAIlEgwA0AntkNatqogyxRmu/eBwoeef0vL0NIuy28cLxOhJiPEF0FAAAAABYAFEvA3l8k2Mluo/gq25aXUGgLnbIKAAAAAAABAN4CAAAAAAEBqsspoMcDbEYKlv0NG3pKV2EwFRztVuHxmyTO+T+YjWQAAAAAAP7///8CgJaYAAAAAAAWABQFIDWTsJHznOA0j0rw5bMdG8s5KdjIOe0AAAAAFgAUU4gJFkaU0xR4CwWuADVHMEG+i4wCRzBEAiAmBS3TFeowHToqqhlyGg8pkGkZVNrd9Fv2iRe2LsSrIwIgQE2haD4jUKPj72oYHJR6al/Nyir6/0yvr2Q/y53br50BIQL+FyFEhKNsYJ7InhYeevCgeMD/vkKXpucUlyJ15VOB334AAAABAR+AlpgAAAAAABYAFAUgNZOwkfOc4DSPSvDlsx0byzkpAQMEAQAAACIGAk+BQfWY797qyLNWalHih2NgeAaH/VSOJI1iSPUbKrW9GAAAAABUAACAAAAAgAAAAIAAAAAAAAAAAAABASvA4eQAAAAAACJRIC84W2gKpgQVStK4bwsZb9I/AqSiW2EJSqU4eTkEwr4jIgYDiBNODl35z2egQMMSgN6jPbVZ62RFfgWeWX6QJVU6SZEYAAAAAFYAAIAAAACAAAAAgAAAAAAAAAAAIRaIE04OXfnPZ6BAwxKA3qM9tVnrZEV+BZ5ZfpAlVTpJkRkAAAAAAFYAAIAAAACAAAAAgAAAAAAAAAAAAAEA3gIAAAAAAQFrR+fYyt6//wJIAJHdH5XBgOirx3uaG6FZccHY+VCnKAEAAAAA/v///wIALTEBAAAAABYAFC3mKCdniaZv0yFVslbfNBaCHWVw3AvWCQEAAAAWABRQLHkZcpMLiJVmDMyF5moVXqfA1QJHMEQCIGQvfqxH31uhus6Lk95EEw5hVXmLNl5CDQ3nXvJ8aX25AiAhWnm7WE7j9oUxiOZwwx/TDf4EZ4cfhO2m4nmUDar4xQEhAmbDYL/O9TVK1Ci8z+scDkTTT+8BKbn0x/CudQB/7FhwfgAAAAEBHwAtMQEAAAAAFgAULeYoJ2eJpm/TIVWyVt80FoIdZXABAwQBAAAAIgYDkU/08WOboGpxZkYXfVVNXKrBS7A1a4Uvz8XwHWzzfjYYAAAAAFQAAIAAAACAAAAAgAAAAAABAAAAAAEBK8JurgIAAAAAIlEgwA0AntkNatqogyxRmu/eBwoeef0vL0NIuy28cLxOhJgiBgJFvsNHfjo+XIZgEfPyUSrsnOyJWsg7wZZNN1SoIT8R8hgAAAAAVgAAgAAAAIAAAACAAAAAAAEAAAAhFkW+w0d+Oj5chmAR8/JRKuyc7IlayDvBlk03VKghPxHyGQAAAAAAVgAAgAAAAIAAAACAAAAAAAEAAAAAAAA=

Then I added the change output and the 546 sats to send 89985713 sats to the same P2TR address:

cHNidP8BANkCAAAABKL8nNtV6rdlliID0zj3d0il+gkr5B8BWk4u0ST8lFMuAAAAAAAAAAAA39rFflQzhHIZLyEiybvHV4PcPeT4jIkKSIxjeR89PloAAAAAAAAAAADouZgo7pEglYdiqOKjRH7JVc1mXWRyT4YgGoMxwlXD5QAAAAAAAAAAAAF34Eb/J9G6akhqZ163e3aH/U8u+bk4NG+QJOkbnqfmAAAAAAAAAAAAAbESXQUAAAAAIlEgwA0AntkNatqogyxRmu/eBwoeef0vL0NIuy28cLxOhJgAAAAAAAEA3gIAAAAAAQGqyymgxwNsRgqW/Q0bekpXYTAVHO1W4fGbJM75P5iNZAAAAAAA/v///wKAlpgAAAAAABYAFAUgNZOwkfOc4DSPSvDlsx0byzkp2Mg57QAAAAAWABRTiAkWRpTTFHgLBa4ANUcwQb6LjAJHMEQCICYFLdMV6jAdOiqqGXIaDymQaRlU2t30W/aJF7YuxKsjAiBATaFoPiNQo+PvahgclHpqX83KKvr/TK+vZD/LnduvnQEhAv4XIUSEo2xgnsieFh568KB4wP++Qpem5xSXInXlU4HffgAAAAEBH4CWmAAAAAAAFgAUBSA1k7CR85zgNI9K8OWzHRvLOSkBAwQBAAAAIgYCT4FB9Zjv3urIs1ZqUeKHY2B4Bof9VI4kjWJI9Rsqtb0YAAAAAFQAAIAAAACAAAAAgAAAAAAAAAAAAAEBK8Dh5AAAAAAAIlEgLzhbaAqmBBVK0rhvCxlv0j8CpKJbYQlKpTh5OQTCviMiBgOIE04OXfnPZ6BAwxKA3qM9tVnrZEV+BZ5ZfpAlVTpJkRgAAAAAVgAAgAAAAIAAAACAAAAAAAAAAAAhFogTTg5d+c9noEDDEoDeoz21WetkRX4Fnll+kCVVOkmRGQAAAAAAVgAAgAAAAIAAAACAAAAAAAAAAAAAAQDeAgAAAAABAWtH59jK3r//AkgAkd0flcGA6KvHe5oboVlxwdj5UKcoAQAAAAD+////AgAtMQEAAAAAFgAULeYoJ2eJpm/TIVWyVt80FoIdZXDcC9YJAQAAABYAFFAseRlykwuIlWYMzIXmahVep8DVAkcwRAIgZC9+rEffW6G6zouT3kQTDmFVeYs2XkINDede8nxpfbkCICFaebtYTuP2hTGI5nDDH9MN/gRnhx+E7abieZQNqvjFASECZsNgv871NUrUKLzP6xwORNNP7wEpufTH8K51AH/sWHB+AAAAAQEfAC0xAQAAAAAWABQt5ignZ4mmb9MhVbJW3zQWgh1lcAEDBAEAAAAiBgORT/TxY5uganFmRhd9VU1cqsFLsDVrhS/PxfAdbPN+NhgAAAAAVAAAgAAAAIAAAACAAAAAAAEAAAAAAQErwm6uAgAAAAAiUSDADQCe2Q1q2qiDLFGa794HCh55/S8vQ0i7LbxwvE6EmCIGAkW+w0d+Oj5chmAR8/JRKuyc7IlayDvBlk03VKghPxHyGAAAAABWAACAAAAAgAAAAIAAAAAAAQAAACEWRb7DR346PlyGYBHz8lEq7JzsiVrIO8GWTTdUqCE/EfIZAAAAAABWAACAAAAAgAAAAIAAAAAAAQAAAAAA

Which I was then able to finalize into:

02000000000104a2fc9cdb55eab765962203d338f77748a5fa092be41f015a4e2ed124fc94532e000000000000000000dfdac57e54338472192f2122c9bbc75783dc3de4f88c890a488c63791f3d3e5a000000000000000000e8b99828ee9120958762a8e2a3447ec955cd665d64724f86201a8331c255c3e50000000000000000000177e046ff27d1ba6a486a675eb77b7687fd4f2ef9b938346f9024e91b9ea7e600000000000000000001b1125d0500000000225120c00d009ed90d6adaa8832c519aefde070a1e79fd2f2f4348bb2dbc70bc4e84980247304402205ca89f8af39b879c8fdb99e4a4a9041270a9dc331ca238de1e77d6eb3637163f0220078f93200be0ba7c34113167ee76a49698c7afac2f823eb9148005198ac276a40121024f8141f598efdeeac8b3566a51e2876360780687fd548e248d6248f51b2ab5bd01408fcc5ef315fb74f636a2f4b9768f5cfadf92706b97180e37c4eb8d258f79594e41c8054139a54afe841cdde47f8d9dfc2f96a06413bb6917e43e37449df3002c02483045022100b7616e259b9d1ce929ac3ac723770b55abefbca79b5fd2cad9e88931073b5255022014f68dfa30ce0f1eeab439cfdafbb2851ef077225bb79d8b8776b0eb83e7cabc012103914ff4f1639ba06a716646177d554d5caac14bb0356b852fcfc5f01d6cf37e360140f4cc0cb26c34cede77c81497092029334a5e4ccbba131931edfe93efd5ee000ef79c660472d77ece349a75c2e5cdda328a0bb2becafcf38c0047b19c6c025b3200000000

Do you maybe have an example of a PSBT that is failing? What inputs are you using?

[alexbosworth]

Sure I can reproduce it on regtest

Create a template PSBT:

  inputs: [
    {
      lock_expires_at: '2022-10-25T15:20:05.000Z',
      lock_id: 'ede19a92ed321a4705f8a1cccc1d4f6182545d4bb4fae08bd5937831b7e38f98',
      transaction_id: '1452c94e21171d250411d10fa830ef2853c5d36b4965e88bdcc46cea13b0b113',
      transaction_vout: 0
    }
  ],
  outputs: [
    {
      is_change: false,
      output_script: '512089c395403dd08bde75ee2d6b6f5c870f5b474deb66e7d915c54ce4adffcf6be2',
      tokens: 546
    },
    {
      is_change: true,
      output_script: '0014db0c7f9d08fa6ed357657880938aacf03ad3f70b',
      tokens: 4999999025
    }
  ],

The PSBT is:

70736274ff01007d020000000113b1b013ea6cc4dc8be865496bd3c55328ef30a80fd11104251d17214ec9521400000000000000000002220200000000000022512089c395403dd08bde75ee2d6b6f5c870f5b474deb66e7d915c54ce4adffcf6be231ee052a01000000160014db0c7f9d08fa6ed357657880938aacf03ad3f70b000000000001012b00f2052a0100000022512089c395403dd08bde75ee2d6b6f5c870f5b474deb66e7d915c54ce4adffcf6be22206028f4544728f9ded35bbb1543fdb78f7c3a80564b1de897d42bb7f30f5e2f0434b1800000000560000800000008000000080000000000000000021168f4544728f9ded35bbb1543fdb78f7c3a80564b1de897d42bb7f30f5e2f0434b1900000000005600008000000080000000800000000000000000000000

Now that I have that, I will make the solo output with the same input but an aggregate output (with FundPsbt):

  fee_tokens_per_vbyte: 3,
  inputs: [
    {
      transaction_id: '1452c94e21171d250411d10fa830ef2853c5d36b4965e88bdcc46cea13b0b113',
      transaction_vout: 0
    }
  ],
  outputs: [
    {
      address: 'bcrt1p38pe2spa6z9aua0w944k7hy8pad5wn0tvmnaj9w9fnj2ml70d03q0cucwp',
      tokens: 4999999571
    }
  ]

This produces the error (on LND 0.15.3)

[Impa10r]

Hello. Is the bug being looked at? I am on 0.17.5 and cannot spend UTXOs exactly on Testnet. I learned to keep haircutting the output amount by a few sats until the error goes away.

[guggero]

This should be fixed with v0.18.0 which is coming soon. Was fixed with https://github.com/lightningnetwork/lnd/pull/8378.

@alexbosworth since this explicitly mentions FundPsbt, do you think we can close this issue?

[alexbosworth]

I didn't play with the changes, but I read it to understand that the old behavior would remain the same and the new behavior would only trigger via the new arguments?

[guggero]

Correct, the new behavior only triggers via the new lncli wallet psbt fundtemplate command (which corresponds to the new coin_select field in the FundPsbt RPC). I didn't want to break existing users or assumptions by modifying the existing behavior. But since a method for creating a 1-in-1-out PSBTs now exists, do you agree with closing this issue?

[alexbosworth]

If it's fixed sure I just didn't get a chance to test it locally yet

[guggero]

Fixed by https://github.com/lightningnetwork/lnd/pull/8378.

[Impa10r]

Correct, the new behavior only triggers via the new lncli wallet psbt fundtemplate command (which corresponds to the new coin_select field in the FundPsbt RPC).

Hello. Could you explain how to use this new coin_select via RPC?

walletrpc.FundPsbtRequest has a new Template available FundPsbtRequest_CoinSelect with one field CoinSelect:

type PsbtCoinSelect struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	// The template to use for the funded PSBT. The template must contain at least
	// one non-dust output. The amount to be funded is calculated by summing up the
	// amounts of all outputs in the template, subtracting all the input values of
	// the already specified inputs. The change value is added to the output that
	// is marked as such (or a new change output is added if none is marked). For
	// the input amount calculation to be correct, the template must have the
	// WitnessUtxo field set for all inputs. Any inputs already specified in the
	// PSBT must already be locked (if they belong to this node), only newly added
	// inputs will be locked by this RPC.
	Psbt []byte `protobuf:"bytes,1,opt,name=psbt,proto3" json:"psbt,omitempty"`
	// Types that are assignable to ChangeOutput:
	//
	//	*PsbtCoinSelect_ExistingOutputIndex
	//	*PsbtCoinSelect_Add
	ChangeOutput isPsbtCoinSelect_ChangeOutput `protobuf_oneof:"change_output"`
}

But where do I get an existing Psbt to provide to it? I invoke FundPsbt to create a Psbt like this (with one output):

res, err := cl.FundPsbt(ctx, &walletrpc.FundPsbtRequest{
		Template: &walletrpc.FundPsbtRequest_Raw{
			Raw: &walletrpc.TxTemplate{
				Inputs:  inputs,
				Outputs: outputs,
			},
		},
		Fees: &walletrpc.FundPsbtRequest_SatPerVbyte{
			SatPerVbyte: feeRate,
		},
		MinConfs:         int32(1),
		SpendUnconfirmed: false,
		ChangeType:       walletrpc.ChangeAddressType_CHANGE_ADDRESS_TYPE_P2TR,
	})

...but it fails unless I haircut the output amount. What am I missing? Thanks!!

[guggero]

You need to specify Template: &walletrpc.FundPsbtRequest_CoinSelect instead of _Raw. See the example in the integration test: https://github.com/lightningnetwork/lnd/blob/036a5a946cfe668199855c66c1ccb2735386d1dd/itest/lnd_psbt_test.go#L1246

The easiest way to create the template PSBT is probably to create it manually from exactly the inputs and outputs you want.

[Impa10r]

You need to specify Template: &walletrpc.FundPsbtRequest_CoinSelect instead of _Raw. See the example in the integration test:

https://github.com/lightningnetwork/lnd/blob/036a5a946cfe668199855c66c1ccb2735386d1dd/itest/lnd_psbt_test.go#L1246

The easiest way to create the template PSBT is probably to create it manually from exactly the inputs and outputs you want.

Thanks, but this "easiest way" is over my head. Too ingrained into testing environment.

I would appreciate a simple example with one utxo as input, one address as output, a fee rate given as a parameter and the resulting amount paid being the utxo amount less the fee.

[guggero]

https://github.com/lightningnetwork/lnd/blob/9f9d1c9e0b03fdaef9daec634a72f773e69be1ec/itest/lnd_psbt_test.go#L1080-L1098 (minus the second output of course)

And then this:

	var buf bytes.Buffer
	err := packet.Serialize(&buf)
	require.NoError(t, err)

	cs := &walletrpc.PsbtCoinSelect{
		Psbt: buf.Bytes(),
		ChangeOutput = &walletrpc.PsbtCoinSelect_ExistingOutputIndex{
			ExistingOutputIndex: 0,
		}
	}

	fundResp := node.RPC.FundPsbt(&walletrpc.FundPsbtRequest{
		Template: &walletrpc.FundPsbtRequest_CoinSelect{
			CoinSelect: cs,
		},
		Fees: &walletrpc.FundPsbtRequest_SatPerVbyte{
			SatPerVbyte: 50,
		},
	})

[Impa10r]

Thank you. This was helpful. The template does not work as I expected. It adds a random additional UTXO to inputs, and sends the total to the single output address. Not a good behavior when sending funds to a third party. I want to spend exactly the selected UTXOs with the fees being deducted from their total amount.

I resorted to using FundPsbtRequest_Raw first with zero outputs, to make it construct one change output. Then I decode that psbt with bitcoin core API to extract the total amount of fees. After that I reduce my sendAmount by that amount of fees and use FundPsbt with manually constructed PSBT.

Pretty much what I did before, but no need to solve for additional haircuts until FundPsbtRequest_Raw funds successfully. Not very elegant, but it works. How can I extract the amount of fees or output value from pbst bytes using btcd?

[guggero]

I think you can achieve what you want in one step by specifying the input you want and using 0 as the output amount, using the ExistingOutputIndex: 0.

[Impa10r]

I think you can achieve what you want in one step by specifying the input you want and using 0 as the output amount, using the ExistingOutputIndex: 0.

0 amount did not work, but 1 sat indeed achieved what I wanted. thank you!

[alexbosworth]

I also created a test case for this and found the changes to be working