Fixes the total num of payments returned by QueryPayments when there are dates restrictions

[MPins]

Fixes #8530

Change Description

The function QueryPayments was not considering the dates restriction when counting the total num of payments.

Steps to Test

lncli listpayments --creation_date_start 1650000000 --count_total_payments --creation_date_end 1650000001 | grep total_num_payments

This command should return 0 as there are no payments between the date start and the date end.

Pull Request Checklist

Testing

• [ ] Your PR passes all CI checks.
• [ ] Tests covering the positive and negative (error paths) are included.
• [ ] Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

• [ ] The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
• [ ] Commits follow the Ideal Git Commit Structure.
• [ ] Any new logging statements use an appropriate subsystem and logging level.
• [ ] Any new lncli commands have appropriate tags in the comments for the rpc in the proto file.
• [ ] There is a change description in the release notes, or [skip ci] in the commit message for small changes.

📝 Please see our Contribution Guidelines for further guidance.

[coderabbitai[bot]]

[!IMPORTANT]

Review skipped

Auto reviews are limited to specific labels.

Labels to auto review (1)

• llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[feelancer21]

@MPins I don't think your fix solves the issue. Using len caps the number of payments at max_payments, which contradicts the definition of the flag count_total_payments. You need to count all payments that are in the database and within the specified date range.

https://github.com/lightningnetwork/lnd/blob/a9655357ca697a01997b941b16be31db611cbf5c/lnrpc/lightning.proto#L4178-L4184

[MPins]

independent of the max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

[feelancer21]

independent of the max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

I am also beginner. As far I understand the code, the maximum length of res.Payments is restricted to MaxPayments because of the paginator. And hence it is not independent of max_payments.

Edit: I made a test on testnet lncli listpayments --count_total_payments --max_payments 1. And total_num_payments changed to 1 now. I think it is not as expected.

[MPins]

independent of the max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

I am also beginner. As far I understand the code, the maximum length of res.Payments is restricted to MaxPayments because of the paginator. And hence it is not independent of max_payments.

Edit: I made a test on testnet lncli listpayments --count_total_payments --max_payments 1. And total_num_payments changed to 1 now. I think it is not as expected.

You right! Thanks! I'll be working on it.

[MPins]

independent of the max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

I am also beginner. As far I understand the code, the maximum length of res.Payments is restricted to MaxPayments because of the paginator. And hence it is not independent of max_payments. Edit: I made a test on testnet lncli listpayments --count_total_payments --max_payments 1. And total_num_payments changed to 1 now. I think it is not as expected.

You right! Thanks! I'll be working on it.

@yyforyongyu now the correction is considering the point indicate by @feelancer21, when the request comes with --count_total_payments and --max_payments causing pagination. Ready to go, thank you both!

[feelancer21]

Thanks. I see some redundant code now, reading the payment and checking the criteria are mainly the same in accumulatePayments and countFn. You should check if it is possible to extract this part in a seperate functions processing the steps of reading and checking.

[MPins]

Thanks. I see some redundant code now, reading the payment and checking the criteria are mainly the same in accumulatePayments and countFn. You should check if it is possible to extract this part in a seperate functions processing the steps of reading and checking.

Yes, thats make sense. Done!

[MPins]

Linter failed, and a nit in the method name. Could you add a release note to 0.18.3? Then it should be good to go.

Thanks!

The linter failed because there was a missing blank line before a 'return'. I can see that in others parts of the code, is there an specific case where the blank line is a must?

The release note could be the following:

• Fixed a bug when RPC call is made to list payments counting the total number of payments.

Should I update the release note file, or you do?

[yyforyongyu]

The linter failed because there was a missing blank line before a 'return'. I can see that in others parts of the code, is there an specific case where the blank line is a must?

It's needed for readability. The linter only checks new code based on a configured look-ahead commit hash. That's probably why you'd see some other parts still use this pattern.

Should I update the release note file, or you do?

Yeah it's usually the author who does the update. You'll find the right section and add the note, also update the author list in the end of the release note file.

[yyforyongyu]

Approved CI run

[MPins]

@yyforyongyu just commited the release notes 0.18.3 changes. Thank you!

[MPins]

@yyforyongyu I did not understand the lint fail!! Any Tip?

[yyforyongyu]

hmm this has failed in two of the itests,

--- FAIL: TestLightningNetworkDaemon/tranche02/45-of-159/bitcoind/list_payments (22.88s)
        --- FAIL: TestLightningNetworkDaemon/tranche02/45-of-159/bitcoind/list_payments/payment_exact_start_date (0.01s)

harness_rpc.go:97: 
        	Error Trace:	/home/runner/work/lnd/lnd/lntest/rpc/harness_rpc.go:97
        	            				/home/runner/work/lnd/lnd/lntest/rpc/lnd.go:187
        	            				/home/runner/work/lnd/lnd/itest/lnd_payment_test.go:286
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = error rolling back tx: conn busy
        	Messages:   	Alice: failed to call ListPayments

Think it's related?

[MPins]

list_payments/payment_exact_start_date

I have run the tests after the change and they are running properly!

Running integration tests with btcd backend. rm -rf itest/.log itest/.logs-; date qua 31 jul 2024 18:45:10 -03 EXEC_SUFFIX= scripts/itest_part.sh 0 1 -test.run="TestLightningNetworkDaemon/tranche./.-of-././list_payments" -test.timeout=180m /home/pins/Projects/lnd/itest/itest.test -test.v -test.run=TestLightningNetworkDaemon/tranche./.-of-././list_payments -test.timeout=180m -logoutput -logdir=.logs-tranche0 -lndexec=/home/pins/Projects/lnd/itest/lnd-itest -btcdexec=/home/pins/Projects/lnd/itest/btcd-itest -splittranches=1 -runtranche=0 === RUN TestLightningNetworkDaemon harness_setup.go:25: Setting up HarnessTest... harness_setup.go:38: Prepare the miner and mine blocks to activate segwit... harness_setup.go:46: Connecting the miner at 127.0.0.1:10446 with the chain backend... harness.go:313: Setting up standby nodes Alice and Bob... harness.go:352: Finished the setup, now running tests... === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_start_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_start_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_end_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_end_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_end_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_and_end_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_start_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_start_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_end_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_end_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_end_date === RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_and_end_date === NAME TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments harness.go:440: finished test: list_payments, start height=440, end height=447, mined blocks=7 === NAME TestLightningNetworkDaemon lnd_test.go:155: =========> tests finished for tranche: 0, tested 156 cases, end height: 440 --- PASS: TestLightningNetworkDaemon (90.83s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments (19.19s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_start_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_start_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_end_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_end_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_end_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_and_end_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_start_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_start_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_end_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_end_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_end_date (0.00s) --- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_and_end_date (0.00s) PASS

[MPins]

sue - what if a user actually wants to know the total num of payments instead of the ranged payments?

About it is an issue or not, I have the following opinion: 1 - If the user want the total num of payments with no date restrictions, why user would set the date range? It is counter intuitive. 2 - Without this fix there is no way to the user know the total num of payments in a date range. 3 - In the text you highlighted it says "independent of the max_payments parameter" not independent of date range.

[guggero]

I'm not sure how to proceed here...

It sounds to me like the expectation for what the --count_total_payments actually does is different between the author of the issue and the reviewers.

Maybe what we need is a new flag, --count_payments_in_range that does the new count as implemented currently but warns the user about the potential performance impact.

Then we can change the description of the --count_total_payments flag to say "counts all payments in the database independent of any filter/range criteria given".

[yyforyongyu]

Thanks a lot for the input so far @MPins ! I'd recommend pausing the development of this PR at the moment as we will soon bring native SQL support for payments in 0.19.0 - maybe we can find a more efficient way to perform the queries by then.

[lightninglabs-deploy]

@mpins, remember to re-request review from reviewers when ready

[yyforyongyu]

!lightninglabs-deploy mute

[MPins]

Thanks a lot for the input so far @MPins ! I'd recommend pausing the development of this PR at the moment as we will soon bring native SQL support for payments in 0.19.0 - maybe we can find a more efficient way to perform the queries by then.

@bhandras @yyforyongyu and @saubyk - do you think it’s a good time to resume work on this PR?

[saubyk]

@bhandras @yyforyongyu and @saubyk - do you think it’s a good time to resume work on this PR?

don't think so. Our plans for the sql schema for payments moved to 0.20, so it would be best to work on this after