🚀 Add bank account transaction integration

[alexanderwassbjer]

Implemented an integration that uses GoCardless api to fetch bank account transactions. Then when you want to add an expense you can either fill it manually or you can pick from your transactions and it will automatically be added as an expense. There is also a function to "multi add" multiple transactions at the same time. If a transaction already has been added to a group you cant add it once again. GoCardless page

Other changes:

• Fixed dark/light mode.
• Fixed "total expenses" which didn't excluded deleted expenses.
• Added a clear field button to add expense page.
• Changed to on addExpense function redirect link.
• Added the new feature to start page.
• Added information about the new feature to readme.

Screenshots:

[alexanderwassbjer]

@KMKoushik :)

[alexanderwassbjer]

I have also added i18n support for this feature in another PR that is based on this PR: https://github.com/oss-apps/split-pro/pull/89. If this gets merged I can open a PR with i18n support for this aswell.

https://github.com/alexanderwassbjer/split-pro/tree/feature/i18n-support

[KMKoushik]

@alexanderwassbjer changed the base to v2!

[alexanderwassbjer]

this looks good overall with some comments. please lemme know what you think!

tysm for this PR really appreciate it. this is really awesome!

think it's best to merge in v2, test thoroughly and release it with other features! (i'll create a new branch)

Awesome! Iam happy you liked it. I will go over it again, check your comments and fix some other stuff.

It would be nice if it was released in v2!

[KMKoushik]

Awesome! Iam happy you liked it. I will go over it again, check your comments and fix some other stuff.

It would be nice if it was released in v2!

ofcourse this is going to be star feature!

[alexanderwassbjer]

@alexanderwassbjer Some other improvement ideas:

• Encrypt the cache for the transactions.
• Add multiple bank accounts for different providers.

[KMKoushik]

i'll have to think about this over the weekend!

[alexanderwassbjer]

@KMKoushik If this is hard to solve or if its because we are limited to the free tier of gocardless, maybe we can have so its for now only possible to use it if you self host?

I mean it's free up to 50 connections per account. Which are more than enough for self hosted ones. https://gocardless.com/bank-account-data/

[gravelfreeman]

The issue with gocardless is that it's mostly supporting European banks. Plaid have a better pricing and has more banks available all across the world. It would be preferred to replace gocardless with Plaid.

https://plaid.com/docs/api/

[Silther]

It would be preferred to replace gocardless with Plaid.

Plaid supports almost no European banks, so you would have to add both.

[alexanderwassbjer]

@gravelfreeman @Silther I think it’s better to start with gocardless, otherwise the PR will just sit there. Adding more providers like plaid in the future isn’t that complicated. This is mainly for those who are self-hosting. Finding a completely free solution that works for all users is hard.

[alexanderwassbjer]

@krokosik What do you think?

[krokosik]

I agree with the approach of adding one provider first (gocardless since it is already implemented) and then someone can add others. I'm sorry I did not look at this PR so far, but have been busy with more core issues since I took over as maintainer.

Recently I have been on vacation a lot and did not finish release 1.5, which I also hope to adjust now to finish up the core issues in my opinion:

• full splitwise import
• splitpro import for migrations
• separating total and friend balances
• minor but annoying UX issues

Once 1.5 is released you can rebase your PR and I can review it then.

[alexanderwassbjer]

I agree with the approach of adding one provider first (gocardless since it is already implemented) and then someone can add others. I'm sorry I did not look at this PR so far, but have been busy with more core issues since I took over as maintainer.

Recently I have been on vacation a lot and did not finish release 1.5, which I also hope to adjust now to finish up the core issues in my opinion:

• full splitwise import
• splitpro import for migrations
• separating total and friend balances
• minor but annoying UX issues

Once 1.5 is released you can rebase your PR and I can review it then.

@krokosik Sounds good, and I really like your approach! I’ll finalize things once 1.5 is released. Please just let me know if you need any help along the way.

[krokosik]

Great! Thank you for not losing interest in this feature, I know it's been a while.

In the meantime you can help with some 1.5 milestone features :)

[alexanderwassbjer]

@krokosik Of course, I think it's a really helpful feature.

Do you have a detailed list of tasks for the 1.5 that I can check? :)

[alexanderwassbjer]

@krokosik I have made this PR up to date with the latest code in main and it's working! And I also did some improvements to the code so it's easier to add more banking providers in the future 😊

[alexanderwassbjer]

@krokosik Its ready for review.

Would also like someone to look at the translations for the other languages.

[alexanderwassbjer]

I left some comments as an initial review from just looking at the code. For a more in depth one, I will need to get GoCardless myself.

I am also concerned about the cache never being deleted from the db. Something like pg-cron will be required. It would be great if you could also create some Typescript API for DB caching, as it would also come in handy with currency rates.

Finally, I would still hold on resolving conflicts until 1.5.0 stable is released. For example, we merged a Portugese locale now, which also needs to be updated with new keys and there may be more bug fixes on the way.

Thanks for your review! Good points. Best would be to signup for GoCardless.

I can take a look at implementing some cron inorder to remove the cache after like a month after its created / updated date. It's a good point.

I will make the changes, then wait for the 1.5 stable release and then resolve the conflicts 😊

[krokosik]

1.5 stable is out :)

Regarding cron there are 2 options that I consider. One is the scheduler in pg-boss, which I would like to add for running background tasks in a separate node process. It does have cron, but I haven't read into the tradeoffs of using it versus pg-cron.

The cache policy should be configurable, one month seems like a lot of transactions to me.

[alexanderwassbjer]

1.5 stable is out :)

Regarding cron there are 2 options that I consider. One is the scheduler in pg-boss, which I would like to add for running background tasks in a separate node process. It does have cron, but I haven't read into the tradeoffs of using it versus pg-cron.

The cache policy should be configurable, one month seems like a lot of transactions to me.

Awesome with 1.5! Then we will try to merge this PR.

I haven’t read much about pg-boss, but I’m considering either that or pg-cron. My gut feeling is that having the cron jobs as a database extension with pg-cron seems more stable than handling them in code with pg-boss — though I might be wrong.

EDIT: The cache policy for manage is 1 day. But 1 month is the getTransactions from GoCardless interval (dateFrom, dateTo). Maybe that should be configurable?

[alexanderwassbjer]

@krokosik I have pushed some new code where all your comments are fixed and some other stuff.

Whats left to do is:

• Use your fork of the nordigen-node
• Implement some cron for pg to remove cache older then like 2 days.

I would like your thoughts on the cron stuff.

[krokosik]

Regarding cron, I will read more about it next week, but so far, it is unclear to me how well pg-cron integrates with Prisma. My initial lookup showed that Postgres extensions are supported as a preview feature, but I have no idea how well. I would also prefer to use pg-cron as a more battle tested and stable solution, but if it's a nightmare in Prisma (which is often the case with advanced DB features) we can skip that.

On the other hand we will use pg-boss anyway. Question is how extensively. If we could have both that would be great. If you could add pg-cron as a proof of concept that would certainly speed things up 🚀

[alexanderwassbjer]

Regarding cron, I will read more about it next week, but so far, it is unclear to me how well pg-cron integrates with Prisma. My initial lookup showed that Postgres extensions are supported as a preview feature, but I have no idea how well. I would also prefer to use pg-cron as a more battle tested and stable solution, but if it's a nightmare in Prisma (which is often the case with advanced DB features) we can skip that.

On the other hand we will use pg-boss anyway. Question is how extensively. If we could have both that would be great. If you could add pg-cron as a proof of concept that would certainly speed things up 🚀

Indeed! I’ve seen that this is still a preview feature, and at the moment Prisma only supports a limited set of extensions. https://www.prisma.io/docs/postgres/database/postgres-extensions Quote:

Another option would be to manually install pg_cron through a migration, but that approach isn’t really recommended either. And we would have to install the pg_cron inside the psql container in docker in order to make the migration to run. https://github.com/prisma/prisma/issues/12308

I will have a look at implementing pg-boss 👍

EDIT: I'am not sure how our self-hosters is running their instance of split-pro. But Netlify, Vercel for example don't keep workers alive, like pg-boss or node-cron in a separate node instance. Instead they recommend using their own way of managing cron jobs. The self-hosters that run in docker on a local server, my self included would probably not have any issues with pg-boss being killed.

[alexanderwassbjer]

@krokosik I have added a proof of concept of the pg_cron. It works good without any problems. 3e873a0 I have some findings:

• Could not add pg_cron and schedule jobs in a prisma migration because it creates a shallow copy which do not have the pg_cron extension available.
• Created a cron setup file which can be called multiple times without duplicated cron jobs being added.
• Added a separate db:cron script which allows us to either run it after the db migration script or separate.
• Had to install pg_cron in the psql instance via a dockerfile which is used in the compose file.
• The pg_cron is logging all of its runs in a table called cron.job_run_details.

I have tested the implementation a bit and I think it's a good approach.

[krokosik]

Awesome work! Will review the changes this week.

[krokosik]

Oof bummer, gocardless does not support individual accounts in my country (Poland) and I don't have a business. We need to figure out a way to test it, are there some other providers I could try?

[alexanderwassbjer]

Oof bummer, gocardless does not support individual accounts in my country (Poland) and I don't have a business. We need to figure out a way to test it, are there some other providers I could try?

Ohno! But do you have like Revolut, because they are supported. Otherwise my plan is to maybe implement the plaid api aswell.

[krokosik]

The requirement to own a business is a requirement for a gocardless account itself. I didn't event get to adding accounts.

[alexanderwassbjer]

The requirement to own a business is a requirement for a gocardless account itself. I didn't event get to adding accounts.

Hmm what? If you go to: https://developer.gocardless.com/bank-account-data/overview

Then it says: "Before you start, please make sure you have acquired a user secret from GoCardless's Bank Account Data portal"

There is a link you can follow.

Which is free and no requirement of a business. That's the way it went and it works good.