New Extension: Scheduled Payouts

[leesalminen]

This is a new extension I'm working on for LNBits: Scheduled Payouts. This extension will allow you to set up recurring payments to LNURL/Lightning Addresses.

[codecov-commenter]

Codecov Report

Merging #881 (1890984) into main (31a51f3) will increase coverage by 0.06%. The diff coverage is 54.70%.

@@            Coverage Diff             @@
##             main     #881      +/-   ##
==========================================
+ Coverage   51.84%   51.90%   +0.06%     
==========================================
  Files         225      232       +7     
  Lines       12351    12638     +287     
==========================================
+ Hits         6403     6560     +157     
- Misses       5948     6078     +130     

Impacted Files	Coverage Δ
lnbits/extensions/scheduledpayments/tasks.py	14.60% <14.60%> (ø)
lnbits/extensions/scheduledpayments/crud.py	41.50% <41.50%> (ø)
lnbits/extensions/scheduledpayments/views_api.py	60.00% <60.00%> (ø)
lnbits/extensions/scheduledpayments/views.py	92.30% <92.30%> (ø)
lnbits/extensions/scheduledpayments/__init__.py	93.75% <93.75%> (ø)
lnbits/extensions/scheduledpayments/models.py	98.38% <98.38%> (ø)
lnbits/extensions/scheduledpayments/migrations.py	100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[leesalminen]

OK, this is the minimum working state. I'll come back around and do some testing.

[leesalminen]

I'm ready for a code review before merge now. Everything is working as expected in my testing.

[leesalminen]

Test server for this PR now live at: https://scheduledpaymentstest.bitcoinjungle.io/

[motorina0]

Test server for this PR now live at: https://scheduledpaymentstest.bitcoinjungle.io/

• currently the server cannot be accessed
• I haven't managed to create Scheduled Payments locally either. It fails with:
• backend error: AssertionError: Newly created schedule couldn't be retrieved
• the entry is added to the DB Table, but it cannot be retrieved

[leesalminen]

Test server for this PR now live at: https://scheduledpaymentstest.bitcoinjungle.io/

• currently the server cannot be accessed
• I haven't managed to create Scheduled Payments locally either. It fails with:
• backend error: AssertionError: Newly created schedule couldn't be retrieved
• the entry is added to the DB Table, but it cannot be retrieved

Yes, I took down the test server for this PR after 30 days. I can bring it back, if needed.

Curious, are you running with Postgres or sqlite? My localhost is on sqlite and cannot recreate this error.

[motorina0]

Curious, are you running with Postgres or sqlite? My localhost is on sqlite and cannot recreate this error.

A: I'm on Postgres, managed to get it working with SQLite

[motorina0]

Some observations:

• is it possible to abuse the cron expression in order to clutter the server with tasks?
• right now it is possible to select a date in the past

[leesalminen]

Curious, are you running with Postgres or sqlite? My localhost is on sqlite and cannot recreate this error.

A: I'm on Postgres, managed to get it working with SQLite

Confirmed I can recreate the issue described when running Postgres. I have no clue why.

[leesalminen]

Some observations:

• is it possible to abuse the cron expression in order to clutter the server with tasks?
• right now it is possible to select a date in the past

The highest resolution the cron expression allows is 1 minute intervals. I suppose it is up for debate whether we should impose a minimum time resolution on schedules. In practice, I can't think of why a user would want to execute a scheduled payment every single minute. Though, on the other hand, it seems like a fairly arbitrary restriction- if a user wants to pay 1 sat every minute, why not?

I suppose a malicious user could create 1000s of schedules to execute every minute. This action wouldn't create 1000s of threads on the server, it would just take longer to process the queue, thus slowing down future payments.

If we do decide to impose a limitation on the cron schedule to prevent abuse, perhaps this should be a configurable setting? For single-user LNBits installations (say, on Umbrel), the abuse potential is basically nil, while on legend.lnbits.com the abuse potential is high. The use cases and risks are very different depending on how LNBits is being used.

[InsaneSplash]

Really would like to see this merged and become a reality...

[leesalminen]

Curious, are you running with Postgres or sqlite? My localhost is on sqlite and cannot recreate this error.

A: I'm on Postgres, managed to get it working with SQLite

Confirmed I can recreate the issue described when running Postgres. I have no clue why.

I'm no longer able to recreate this issue after merging main into this branch, clearing the Postgres db, and re-running the migrations. Would appreciate if someone else could give this branch a try now on Postgres and see if they're having the same issue.

[motorina0]

Would appreciate if someone else could give this branch a try now on Postgres and see if they're having the same issue.

• works fine on Postgres now

[InsaneSplash]

Hi, How things going with this extension :)

[motorina0]

Hi, How things going with this extension :)

• the issue still reproduces for me with a fresh Postgres DB
• @leesalminen does it work for you each time? I cannot figure out what causes this issue.

Note:

• in the near future users will have the option to install extensions from the Admin UI
• this means extensions will live in their own repo

[InsaneSplash]

As many don't use PostGres, could a version without it be released as a start?

[arcbtc]

I want to merge this pr as I really like the extension, however I don't like the dependencies. I think any db issues were because of tz. The two date/time dependencies can be removed and native datetime could be used. The crontools dependency is too underused a library for having in lnbits core https://github.com/dapper91/crontools.

[arcbtc]

Made a cron job task instead that this and other extensions could use: https://github.com/lnbits/lnbits/pull/1179

[bitkarrot]

I want to merge this pr as I really like the extension, however I don't like the dependencies. I think any db issues were because of tz.

From the article linked below:

Python 3.9 introduced zoneinfo into the standard library.

Before zoneinfo, the Python ecosystem relied on pytz for time zones. If you can migrate to zoneinfo, it’s better, as the pytz API has ambiguities, and it bundles its own copy of the tz database that is not always up to date.

Note that on some operating systems, notably Windows, there is no accessible copy of the tz database. In this case you should also install the official tzdata package which zoneinfo will read from.

https://adamj.eu/tech/2021/05/06/how-to-list-all-timezones-in-python/

[dni]

this extension should be its own repo, now after we switched @leesalminen

[supertestnet]

What is the status of this extension? Autopay is still very hard on lightning -- and on bitcoin in general

[motorina0]

What is the status of this extension? Autopay is still very hard on lightning -- and on bitcoin in general

@bitkarrot what is the status of this: https://github.com/lnbits/lnbits/issues/1724

[bitkarrot]

What is the status of this extension? Autopay is still very hard on lightning -- and on bitcoin in general

@bitkarrot what is the status of this: #1724

@motorina0 I'm working on it. Should have something by next week. Sorry for the delay

[dni]

can you refactor it out into its own repo and close this pull request? i think it does not belong inside the lnbits bit repos anymore, right?

[ThomasFarstrike]

@bitkarrot what's the status of #1724 ?

I'm also in need of this, and willing to code on it, so let me know if you would like to split the work to help bring it across the finish line!

[ThomasFarstrike]

can you refactor it out into its own repo and close this pull request? i think it does not belong inside the lnbits bit repos anymore, right?

@motorina0 I'm also interested in this! Are you planning to move it to its own repo? Which URL or repo name do you have in mind?

[ThomasFarstrike]

I'm wondering what the best way forward is here...

From my limited knowledge, it seems that in a perfect world, this extension would make use of the new lnbits/task timed_job(, ) at https://github.com/lnbits/lnbits/pull/1179/files to avoid having an additional thread looping and checking for tasks to run.

But to already get this across the finish line, as it's been in the works for a long time now, the initial version could be released as is, and then later optimized by making use of timed_job.

What's your view on this? I just want recurring payments and I'm willing to get my hands dirty to expedite the process ;-) So let me know what I can do to push this along.

The recurring payments use case I'm thinking of has to do with https://lightningpiggy.com/ - like daily/weekly allowances etc.

[motorina0]

can you refactor it out into its own repo and close this pull request? i think it does not belong inside the lnbits bit repos anymore, right?

@ThomasFarstrike

@motorina0 I'm also interested in this! Are you planning to move it to its own repo?

• yes, it should be its own repo
• name: scheduler

this extension would make use of the new lnbits/task timed_job(, )

• this is too limited as it would not allow task of the form Run this job every Monday at 07:00

the initial version could be released as is, and then later optimized by making use of timed_job.

• an "upgrade" path from timed_job to cron is not possible, we should use with cron from the start
• we have decided that it is OK to add the python-crontab library to the lnbits core, so the extension can use it
• see https://github.com/lnbits/lnbits/issues/1724

[bitkarrot]

@bitkarrot what's the status of #1724 ?

I'm also in need of this, and willing to code on it, so let me know if you would like to split the work to help bring it across the finish line!

I'm still working on this and have an existing framework, DM me on telegram if you want to help out

[ThomasFarstrike]

I can't view the logs of the failed check "migrations / sqlite-to-postgres (3.9, 1.2.1) (pull_request)" as they've been removed for being too old.

In general, I'm struggling a bit to identify the next actions to take on this - please help me out!

@motorina0 I'm also interested in this! Are you planning to move it to its own repo?

* yes, it should be its own repo

* name: `scheduler`

@motorina0 Do you see this extension here existing alongside #1724? Or is #1724 supposed to land at https://github.com/lnbits/scheduler and is it to replace/supersede/deprecate this extension here?

In the former case, perhaps you could create the new repo https://github.com/lnbits/scheduler (or give me moar permissions!) and move the code there (or give me moar permissions!) and then we can re-run the checks there, see whether they're still failing, and have the up-to-date results to view?

In the latter case, I guess the new repo should be created by someone, and then I can collaborate there with @bitkarrot to push this along.