`pkgx --sync` on `main` takes around 1 minute while in `v1.1.6` takes 1 second

[felipecrs]

I noticed this so I just wanted to make you aware (in case you aren't already).

This is because the addition of the cache using sqlite, which didn't exist on 1.1.6.

[dosubot[bot]]

The performance issue with pkgx --sync on the main branch taking around 1 minute, compared to version v1.1.6 taking only 1 second, could indeed be related to the addition of the cache using SQLite ^[1].

To continue talking to Dosu, mention @dosu.

[mxcl]

@dosubot is probs right. The cache takes time to build, though for me it is only a few seconds or I wouldn't be about to publish v1.2 that contains it…

I guess we'll find out if this still occurs for you on release and if so fix that straight away.

[mxcl]

Note that sync is slower but running executables has way, way less overhead. I did the testing months ago but it was 50x or something

[felipecrs]

That is understandable, but I was wondering if something could be done to improve it. I use pkgx very often in ephemeral environments (containers, CI/CD), and taking 1 minute for the sync (which must be done in fresh environments) is kind of big deal.

$ git pull
$ dev
$ deno task run --sync  6.76s user 8.04s system 21% cpu 1:08.82 total

[mxcl]

yeah this is not ok for sure.

The code I wrote is not optimized, but I also don’t know where the slowness is specficially:

https://github.com/pkgxdev/libpkgx/blob/main/src/hooks/useSyncCache.ts

• Potentially the ffi interface to sqlite is slow
• Potentially the sqlite queries are slow
• Potentially the slowdown is entirely in the YAML parsing

I will aim to investigate but not today (likely).

[mxcl]

I did intend to bundle the pre-cached DB with the pantry.tgz that pkgx downloads, but we should aim to make the cache-step fast as first as reasonably possible anyway.

sqlite is potentially not the right solution, though it's accessible and simple.

[mxcl]

just realized that the installation of the sqlite pkg will add seconds for sure. That is step one in caching. It's only 3MB tho.

[amospalla]

Moving the .cache/pkgx folder into a memory based filesystem (like /dev/shm or /tmp if on memory) makes the sync process instant, while on my ssd disk stays writing 120mb/s during several seconds.

[felipecrs]

Thanks for the hint, @amospalla.

@mxcl, https://github.com/pkgxdev/libpkgx/pull/78 fixes it.

[mxcl]

A new pkgx release coming soon.