:sparkles: (api) ability to run bank sync via the API

[MatissJanis]

Allow running 3rd party (gocardless, simplefin) bank-sync via the API package.

This will allow people to run the bank-sync operation in the background on a schedule (i.e. with cron jobs).

Usage:

const api = require('@actual-app/api');

(async () => {
  await api.init({
    // Budget data will be cached locally here, in subdirectories for each file.
    dataDir: '/path/to/budget/file',
    // This is the URL of your running server
    serverURL: 'https://xxxxxxx.fly.dev',
    // This is the password you use to log into the server
    password: 'xxxxxx',
  });

  // This is the ID from Settings → Show advanced settings → Sync ID
  await api.downloadBudget('xxxxxx-xxxx-xxxxx-xxxxx');

  // Run bank sync for a single account
  await api.runBankSync({ accountId: 'xxxxxx-xxxxx-xxx-xxxxx-xxxxxxxx' });

  // Run bank sync for all accounts
  await api.runBankSync();

  await api.shutdown();
})();

Docs: https://github.com/actualbudget/docs/pull/350

[netlify[bot]]

✅ Deploy Preview for actualbudget ready!

Name	Link
🔨 Latest commit	69aa671b80729b9974252a1f140297d2ae9e90dd
🔍 Latest deploy log	https://app.netlify.com/sites/actualbudget/deploys/6644df46114a1a00083ba3a2
😎 Deploy Preview	https://deploy-preview-2683.demo.actualbudget.org
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions[bot]]

Bundle Stats — desktop-client

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
9	4.72 MB → 4.72 MB (-6 B)	-0.00%

Changeset

File	Δ	Size
home/runner/work/actual/actual/packages/loot-core/src/client/actions/account.ts	📉 -6 B (-0.14%)	4.11 kB → 4.1 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

No assets were bigger

Smaller

Asset	File Size	% Changed
static/js/index.js	3 MB → 3 MB (-6 B)	-0.00%

Unchanged

Asset	File Size	% Changed
static/js/indexeddb-main-thread-worker-e59fee74.js	13.5 kB	0%
static/js/resize-observer.js	18.37 kB	0%
static/js/BackgroundImage.js	122.29 kB	0%
static/js/narrow.js	59.81 kB	0%
static/js/usePreviewTransactions.js	790 B	0%
static/js/AppliedFilters.js	20.54 kB	0%
static/js/wide.js	261.78 kB	0%
static/js/ReportRouter.js	1.23 MB	0%

[github-actions[bot]]

Bundle Stats — loot-core

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
1	1.2 MB → 1.2 MB (+431 B)	+0.03%

Changeset

File	Δ	Size
packages/loot-core/src/shared/errors.ts	📈 +122 B (+3.23%)	3.69 kB → 3.81 kB
packages/api/methods.ts	📈 +391 B (+2.98%)	12.81 kB → 13.19 kB
packages/loot-core/src/server/api.ts	📈 +286 B (+1.53%)	18.28 kB → 18.56 kB
packages/loot-core/src/server/main.ts	📈 +43 B (+0.07%)	59.95 kB → 59.99 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

Asset	File Size	% Changed
kcab.worker.js	1.2 MB → 1.2 MB (+431 B)	+0.03%

Smaller

No assets were smaller

Unchanged

No assets were unchanged

[matt-fidd]

When calling runBankSync with no arguments, I'm receiving the below error.

/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:43402
            handlers['accounts-bank-sync'] = async function ({ id }) {
                                                                                             ^

TypeError: Cannot destructure property 'id' of 'undefined' as it is undefined.
    at handlers.accounts-bank-sync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:43402:64)
    at handlers.api/bank-sync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34823:72)
    at runHandler (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:44781:33)
    at Object.send (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:44463:94)
    at send (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/methods.js:31:21)
    at Object.runBankSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/methods.js:58:12)
    at /data/home/matt/cron/actual/bankSync.js:25:12

Node.js v18.19.1

[MatissJanis]

Thanks for testing @matt-fidd ! I must have accidentally removed something when I cleaned up the implementation.

Should be patched now. Give it another go.

[matt-fidd]

Thanks @MatissJanis!

Looks better now!

It appears to hit a problem halfway through my sync where it tries to re sync with the server in the middle of polling gocardless and falls over.

Bank Sync operation for account: ########
  Pulling transactions from GoCardless
  Syncing since 2024-05-01T17:10:28.143Z-0000-b80ed5e91c4802ff 7 (attempt: 0)
  SyncError: SyncError: encrypt-failure
      at Module.encode (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47024:35)
      at _fullSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47705:77)
      at async /data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47611:32 {
    reason: 'encrypt-failure',
    meta: { isMissingKey: true }
  }

From then on, any API interactions are met with the below error.

node bankSync.js
- Connecting
-- Success
- Downloading
Loaded spreadsheet from cache (3136 items)
Syncing since 2024-05-01T17:10:28.143Z-0000-b80ed5e91c4802ff 7 (attempt: 0)
SyncError: SyncError: encrypt-failure
    at Module.encode (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47024:35)
    at _fullSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47705:77)
    at async /data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47611:32
    at async initialFullSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47597:32)
    at async handlers.sync-budget (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:43980:32)
    at async handlers.api/download-budget (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34777:36) {
  reason: 'encrypt-failure',
  meta: { isMissingKey: true }
}
/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34779
                        throw new Error((0, _shared_errors__WEBPACK_IMPORTED_MODULE_2__.getSyncError)(result.error, localBudget.id));
                              ^

Error: We had an unknown problem opening ?My-Finances-5cca4be?.
    at handlers.api/download-budget (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34779:31)

Node.js v18.19.1

and to be able to continue, I have to wipe the cache directory and resync from blank

[matt-fidd]

Some debugging later, the error is thrown here as the keys object is empty.

With the debug lines below added

export function getKey(keyId) {
  if (keyId == null || keys[keyId] == null) {
+    console.log(keyId);
+    console.log(keys);
+    console.log(keys[keyId]);
    throw new Error('missing-key');
  }
  return keys[keyId];
}

The following is printed (master key has been redacted but it is set!)

Loaded spreadsheet from cache (3136 items)                                                                                                                                                                                                                                  
Syncing since 2024-05-01T21:31:22.845Z-0073-9f76373186bb94dd 13 (attempt: 0) 
                                                                                                                                                                                               
***REDACTED KEY***                                                                                                                                                                                                                                       
{}                                                                                                                                                                                                                                                                          
undefined              
                                                                                                                                                                                                                                                     
SyncError: SyncError: encrypt-failure
...

[MatissJanis]

Looks like the e2e encryption is having problems there for you. Did you set the encryption password in the code?

https://actualbudget.org/docs/api/#connecting-to-a-remote-server

await api.downloadBudget('1cfdbb80-6274-49bf-b0c2-737235a4c81f', {
    password: 'password1',
});

[matt-fidd]

I did set the encryption password and it manages to connect and sync a couple of accounts, and then it gets caught up somewhere and empties the keys object in /loot-core/src/server/encryption.ts. I've checked and neither the unloadKey or unloadAllKeys functions have been called.

[matt-fidd]

Done some more digging, I can't force it to happen with any other API functions. If I just sync one bank account over and over it also does not happen, only when calling api.runBankSync();

[MatissJanis]

Thanks for the investigation @matt-fidd . I've split off the patch of this issue into a separate PR: https://github.com/actualbudget/actual/pull/2698

[matt-fidd]

Amazing, thank you @MatissJanis!

[youngcw]

@MatissJanis Is there much more that needs added here or is the ready for review?

[MatissJanis]

@youngcw this Pr depends on #2698. Once that one is merged - I'll open this one back up.

[MatissJanis]

Out for review again.

Would appreciate another round of testing from you @matt-fidd ! 🙇

[matt-fidd]

All looking good!!