New Import source - Budgetbackers Wallet app

[rlesniak]

I wanted to move out of wallet app so I wrote importer. Tested it with a large 700KB CSV file (several years of data) and it worked like a charm.

It relates to #1198

[netlify[bot]]

✅ Deploy Preview for actualbudget ready!

Name	Link
🔨 Latest commit	7f7c583ba6871615813650664e44cacc707d4d28
🔍 Latest deploy log	https://app.netlify.com/sites/actualbudget/deploys/6552194941d94f000851de8c
😎 Deploy Preview	https://deploy-preview-1866.demo.actualbudget.org/
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions[bot]]

Bundle Stats — desktop-client

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
16	2.85 MB → 2.85 MB (+1.85 kB)	+0.06%

Changeset

File	Δ	Size
src/components/manager/ImportWallet.tsx	🆕 +3.88 kB	0 B → 3.88 kB
src/components/manager/Import.js	📈 +1012 B (+20.72%)	4.77 kB → 5.76 kB
src/components/manager/Modals.js	📈 +242 B (+7.27%)	3.25 kB → 3.49 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

Asset	File Size	% Changed
static/js/main.js	1.12 MB → 1.12 MB (+1.85 kB)	+0.16%

Smaller

No assets were smaller

Unchanged

Asset	File Size	% Changed
static/js/460.chunk.js	774.35 kB	0%
static/media/Inter-italic.var.woff2	239.29 kB	0%
static/media/Inter-roman.var.woff2	221.86 kB	0%
static/js/444.chunk.js	156.11 kB	0%
static/js/wide-components.chunk.js	127.08 kB	0%
static/js/231.chunk.js	117.37 kB	0%
static/js/narrow-components.chunk.js	52.15 kB	0%
static/js/reports.chunk.js	35.97 kB	0%
static/js/473.chunk.js	13 kB	0%
static/js/312.chunk.js	12.93 kB	0%
static/js/resize-observer-polyfill.chunk.js	8.88 kB	0%
static/css/main.css	7.41 kB	0%
asset-manifest.json	2.07 kB	0%
index.html	1.66 kB	0%
static/media/browser-server.js	903 B	0%

[github-actions[bot]]

Bundle Stats — loot-core

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
2	2.22 MB → 2.22 MB (+2.57 kB)	+0.11%

Changeset

File	Δ	Size
packages/loot-core/src/server/importers/wallet.ts	🆕 +5.88 kB	0 B → 5.88 kB
packages/loot-core/src/server/importers/index.ts	📈 +56 B (+4.38%)	1.25 kB → 1.3 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

Asset	File Size	% Changed
kcab.worker.js	1.23 MB → 1.23 MB (+2.57 kB)	+0.20%

Smaller

No assets were smaller

Unchanged

Asset	File Size	% Changed
xfo.xfo.kcab.worker.js	1014.46 kB	0%

[youngcw]

Do you have a sanitized file or an example budget to test the import? Code looks pretty good

[rlesniak]

I can attach CSV from my account, is that enough? report_2023-11-02_163422.csv

[youngcw]

ok, that worked as far as I can tell. Is that the full budget?

[rlesniak]

No, its small part of everything. I cant attach full dump from obvious reasons

[youngcw]

As long as its a good representation of a full budget its fine. If you fix those missing tests I can look at approving this

[rlesniak]

However I found some weird behavior. After import, I have to double click on green tick on the right side of list. When I click, sum of accounts changes toward to real value on accounts. It happens only on transfer entries, and not every transfer. Look at the videos

https://github.com/actualbudget/actual/assets/13970783/00c22403-3882-472c-829e-f11d9214d031

https://github.com/actualbudget/actual/assets/13970783/9976d92e-18fa-49df-8bbb-294367543870

Is this something related to my import code or could be Actual issue in general?

[rlesniak]

Any hint @youngcw ?

[jsehnoutka]

Hi @rlesniak !

First of all, thanks for this! I am now trying to get familiar with Actual and switching over to it having also several years worth of Wallet data. Wallet is a very mature product, but one specific critical bug (transfer between accounts is buggy and unfixed for years) eventually got me here...so far seeing great potential in Actual, hope I can soon contribute as well.

I tested your modification and have some questions in mind (I am not able to read code so well yet to figure it out from your commits, sorry)

1. Is there a possibility to deal with for budget/on budget classification for each of the accounts before executing the import itself? (from Wallet POV, I have a few accounts set as "excluded from stats", but still track transactions on them). I am not sure if the Actual import tool even allows to implement this approach (maybe an extra screen like there already is for column selection in the basic import?) As per documentation, Actual doesn't allow to change account classification once it's created.

Workaround - use the import tool, for creating "for budget" accounts, then export csv for respective accounts which should be off budget. Delete on budget account, create new off budget account manually, import transactions from beforehand exported csv.

2. Transfers are not propagating properly, possible need to adjust each transaction by hand after import.

Will try this script after import which should deal with it.

Also throughout the years, I've used the "Note" field instead of "Payee" in Wallet (because Payee note did not have auto-search/index from previous data - simply genius, Budgetbakers!) - this messes up my data quite a lot. Will need to figure some way to clean up the data later...

Again, thanks for your effort, especially since this is essentialy an one-time use case. I would like to hear some details and/or inspiration on your migration strategy from Wallet, as there are some misalignments between Wallet x Actual approach which are not making things easy...

[rlesniak]

Hey @jsehnoutka!

The Wallet CSV file is in fact very much lacking in a lot of information, unfortunately 😞

Is there a possibility to deal with for budget/on budget classification for each of the accounts before executing the import itself?

From what I learned, I assume that we can create some kind of "wizard" after processing file. But it would require much more effort.

Transfers are not propagating properly, possible need to adjust each transaction by hand after import.

Can you elaborate on this? To detect transfers, I look for this information in the csv:

• Find entries with category TRANSFER and minus amount
• Look for adjacent entries that have exact same value but opposite.
• If not found, treat the transfer as a transfer to the payee column. based on this data, determine to/from account name

I do not see any other way to do this.

There is a high risk that importing from the Wallet will require manual tweaking anyway.

[jsehnoutka]

Thanks for the additional info, I found the issue. You're using Wallet in Polish, whereas I am using it in English. This is affecting the data inside csv.

In Polish csv, the transfer can by determined by category "TRANSFER", but in English csv, it's "Transfer, withdraw". I've searched and replaced my csv from "Transfer, withdraw" to "TRANSFER" and afterwards transfers propagated correctly. But the import itself is still not matching at all (I get discrepancy on accounts balances), there is still some error that I am missing, probably not importer related though...

I propose you adjust the importer to use English (as it is this project's language) or, choosing Wallet language would need to be part of the input wizard. To be honest, I don't really see much added value in such a wizard as the workload on it would be significant and benefit not so much.

Also a note to change the Wallet language to whatever is hardcoded into the importer must be mentioned in bold before loading the file into the importer as mismatch in language will cause the import to fail.

See my dummy import (original and adjusted) if it helps:

report_2023-11-09_172649_TRANSFER.csv report_2023-11-09_172649.csv

[jsehnoutka]

Hmm, after further diving into my data, I am confident that the importer completely ignores all transactions in 4 of my accounts, even though these are present in the csv. Only transfers from different (not ignored) accounts get propagated into these 4 "blacklisted" accounts.

This is oddly specific given the fact I have 14 accounts in total. Is it possible that number of accounts considered during the import is high capped to 10?

[rlesniak]

In Polish csv, the transfer can by determined by category "TRANSFER", but in English csv, it's "Transfer, withdraw". I've searched and replaced my csv from "Transfer, withdraw" to "TRANSFER" and afterwards transfers propagated correctly. But the import itself is still not matching at all (I get discrepancy on accounts balances), there is still some error that I am missing, probably not importer related though...

I completely ignored boolean field transfers in csv. I fixed it, now it is clear for the importer no matter what language is used.

This is oddly specific given the fact I have 14 accounts in total. Is it possible that number of accounts considered during the import is high capped to 10?

Its difficult to debug without csv :/ From theretical POV it should work the same for every file. Maybe update your branch to last commit and try again

[jsehnoutka]

Looks like transactions on these blacklisted accounts of mine are being catched out by internal de-dup feature as I figured from some other discussions on the project.

Will look into it more during the weekend but I am very much sure that's the issue - importer non-related, that feature is as robust as it can be after your fix.

I did a quick check and after your latest commit, there were significantly less error transactions but still some, probably caused by de-dup killing off its counterpart. Will investigate during the weekend and prepare some mockup csv if the issue is repeatable but right now, I think it's de-dup what's causing the headache

[youngcw]

I think it's de-dup what's causing the headache

Are you adding two transactions for a transfer? One regular transaction, then a transfer transaction? That could maybe be part of the issue where you are ending up with two, but the transfer one comes second and gets deleted. Maybe?

[jsehnoutka]

Are you adding two transactions for a transfer? One regular transaction, then a transfer transaction? That could maybe be part of the issue where you are ending up with two, but the transfer one comes second and gets deleted. Maybe?

After spending a few more hours on it, I've decided to start with a clean slate on Actual as de-dup is undefeatable. I don't see any way out of it, hundreds of transactions are being de-dupped in my case and I cannot input them manually as docs propose. (I wish the de-dup feature could be toggled off!). On smaller datasets, I suppose it wouldn't be such an issue as manual correction might be feasible.

I will try to double post with both Actual and Wallet as I really want to try Actual. My historic data is important to me, so I probably won't be able to ditch Wallet anytime soon. :(

I completely ignored boolean field transfers in csv. I fixed it, now it is clear for the importer no matter what language is used.

I also found out as Wallet is extremely buggy (the whole reason behind me being here now, lol), that the csv is not always consistent in the transfer field (but category is). Not all transfers in my csv have transfer attribute "true".

I did a quick check and after your latest commit, there were significantly less error transactions

Here I was wrong, it seems that importer doesn't propagate transfers unless category doesn't explicitly say "TRANSFER" and on the same line, transfers say "TRUE" (the second condition kicked in after latest commit).

So before using the importer, I preprocessed the csv data in a way the code below indicates. Then, I was able to import with highest accuracy.

let transactions = [
    { category: "Transfer, withdraw", transfer: "false" },
    { category: "Groceries", transfer: "false" },
    // ... other transaction objects
];

transactions.forEach(transaction => {
    if (transaction.category === "Transfer, withdraw") {
        transaction.transfer = "true";
        transaction.category = "TRANSFER";
    }
});

[github-actions[bot]]

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[rlesniak]

Sorry, I havent much time to work on it. Any new thoughs @jsehnoutka ?

[jsehnoutka]

Hi, my last comment is still valid. Not working properly unless category explicitly match TRANSFER, even after your last PR.

Some server error managed to mess up all my Wallet data and their support suck, so I don't use it anymore. But I still have my old csv that I can use for testing if needed. ;)

[jsehnoutka]

Hi @youngcw, any input I could further work on in order to get this approved?

Although I think it's still a bit glitchy (see my latest comment) - if @rlesniak would be willing to look at it once again, it would be a great addition to Actual. I personally used this PR to finally migrate my Wallet data before it got corrupted. I think given the declining quality of Wallet app, we might be seeing quite many users migrating to Actual. This feature would definitely help with onboarding.

I can work on more thorough testing, preparing sample test csv etc...I just need a little feedback on what's needed.

[youngcw]

@jsehnoutka if you are wanting to take over I would start with getting the github tests passing. There are a few concerns voiced in the comments that I believe still need addressed. Im concerned that if the exports have variance between two countries, how much variation will there be and can that be handled cleanly. Maybe not much and it will be fine.

I believe the api has the option to bypass the de-duplication at import so I think that this could be set up to do that as well.

[jsehnoutka]

Thanks for pointing me in the right direction, this could be a great entry point for me! I will see what I can do.

[github-actions[bot]]

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This PR was closed because it has been stalled for 5 days with no activity.