Why is Nimble slow?

[treeform]

It came to my attention when we added Nim to our company that our CI speed got way, way slower, and I started wondering why. And so I ran a little investigation. So it looks like it takes about 1:51 seconds to install the Nimble packages for Fidget2.

All right, I thought to myself, well, that's a little slow. Let's see if I can replicate what Nimble does with just git commands. So that's what I did. I just git cloned every package and stuck them into nim.cfg, and everything built. And it was way faster! It was only 20 seconds. So what is Nimble doing for that 1:31 seconds extra? Very strange. SAT solving can't take that much. Even like walking the package tree can't take that long.

Then I thought, well, git is actually kind of slow, because you have to do the handshake and do the git:// protocol. What if you just curl & unzip the packages directly? And then I did that, and that was even faster. 16 seconds!

It's like, wow, okay, now we're going. Well, what about if we do it in parallel? In parallel, it's like 3 seconds. Okay, so if you just download zips in parallel, it's 3 seconds. But if you use Nimble, it's 1:51 seconds. Is this true? Am I doing something wrong?

Code here: https://github.com/treeform/fidget2/pull/91/files

* Nimble ..................... 1:51s https://github.com/treeform/fidget2/actions/runs/18803878890/job/53655066175
* Git clone .................... 20s https://github.com/treeform/fidget2/actions/runs/18803878889/job/53655066135
* Curl Zips .................... 16s https://github.com/treeform/fidget2/actions/runs/18803878895/job/53655066174
* Curl Zips in Parallel ......... 3s https://github.com/treeform/fidget2/actions/runs/18803878916/job/53655066170

[Araq]

How does Atlas perform?

[treeform]

Atlas takes 31 seconds.

* Nimble ..................... 1:51s https://github.com/treeform/fidget2/actions/runs/18803878890/job/53655066175
* Atlas ........................ 31s https://github.com/treeform/fidget2/actions/runs/18809587320/job/53668718832
* Git clone .................... 20s https://github.com/treeform/fidget2/actions/runs/18803878889/job/53655066135
* Curl Zips .................... 16s https://github.com/treeform/fidget2/actions/runs/18803878895/job/53655066174
* Curl Zips in Parallel ......... 3s https://github.com/treeform/fidget2/actions/runs/18803878916/job/53655066170

I really liked the Atlas model when it came out, but then all of a sudden then it got complicated and confusing? Then it kind of lost steam and nimble still dominates. Please bring the old atlas back! Or at least improve the docs.

[jmgomez]

What nimble version are you using?

Can you try with --maximumTaggedVersions set to 1 and see if it makes a difference? Anyways, the plan is to focus in perf after we complete the feature set and bug fixing for 1.0.0.

[Araq]

I really liked the Atlas model when it came out, but then all of a sudden then it got complicated and confusing?

How so? It got even simpler...

[treeform]

@jmgomez

Nimble's --maximumTaggedVersions:1 is slightly faster but still 1:44s, maybe just network randomness?

* Nimble ..................... 1:51s https://github.com/treeform/fidget2/actions/runs/18803878890/job/53655066175
* Nimble Tagged 1 ............ 1:44s https://github.com/treeform/fidget2/actions/runs/18812044754/job/53674738501
* Atlas ........................ 31s https://github.com/treeform/fidget2/actions/runs/18809587320/job/53668718832
* Git clone .................... 20s https://github.com/treeform/fidget2/actions/runs/18803878889/job/53655066135
* Curl Zips .................... 16s https://github.com/treeform/fidget2/actions/runs/18803878895/job/53655066174
* Curl Zips in Parallel ......... 3s https://github.com/treeform/fidget2/actions/runs/18803878916/job/53655066170

Run nimble -v
nimble v0.18.2 compiled at 2025-04-22 02:13:48
git hash: couldn't determine git has

[treeform]

@Araq I remember starting out with Atlas when it was really early, and I thought the design was great. First, it created, one single root workspace folder where all the packages were installed. So it was just a giant folder with all the packages, and they were all being git checked out, and they were all ready for development. And I actually used that system for a while, and then I transitioned to using it at Reddit as well. And it was great, it was just install, and everything just made sense. But then, stuff stopped working. I guess it moved into where the Atlas now creates a deps folder into every package that you install, potentially creating multiple copies of all the deps. Maybe like npm? It's just not the way I develop at all. Yes, you can install them for development through these links, but then you have links everywhere. I didn't get it. Like, it was much simpler before, just a big folder and just a single nim.cfg, and that was it. But now you have a folder, and there's deps, and there's links, and there is pins, and there's other stuff. And I think the syntax has changed. Everything broke. I didn't see anyone else use it, so I just kind of gave up on it and went back to Nimble. It took me like 20 min of trial an error to get it to install fidget2 when you asked ... this is not the experience I had when I first started with it.

[Araq]

Fair enough but the old Atlas was always confused about "deps of this package" vs "workspace", most switches had workspace vs project options. It wasn't simple for me after a while. Plus the deps/ directory was what almost everybody else wanted.

It took me like 20 min of trial an error to get it to install fidget2 when you asked

We should definitely improve this experience. :-)

[jmgomez]

@treeform hmm I see, thanks. BTW last released version is 0.20.1 and we have been working for quite a few months now on 1.0.0, which is pretty much a full rewrite of the core while keeping full compat.

[treeform]

@jmgomez The current head actually fails to install the packages! Maybe puts them in a different place? Very strange but it says it downloads them. It takes 1m 23s, so a little faster but very far a way from theoretical speed of 3s.

* Nimble ..................... 1:51s https://github.com/treeform/fidget2/actions/runs/18803878890/job/53655066175
* Nimble Tagged 1 ............ 1:44s https://github.com/treeform/fidget2/actions/runs/18812044754/job/53674738501
* Nimble Head (v0.99.0) ...... 1:23s https://github.com/treeform/fidget2/actions/runs/18819462176/job/53692680748
* Atlas ........................ 31s https://github.com/treeform/fidget2/actions/runs/18809587320/job/53668718832
* Git clone .................... 20s https://github.com/treeform/fidget2/actions/runs/18803878889/job/53655066135
* Curl Zips .................... 16s https://github.com/treeform/fidget2/actions/runs/18803878895/job/53655066174
* Curl Zips in Parallel ......... 3s https://github.com/treeform/fidget2/actions/runs/18803878916/job/53655066170

[jmgomez]

If you fill an issue with a repro I can take a look. We are in the stage of making it correct, which besides a few small things, only requires bug fixing. Then we will make it fast :)

It takes 1m 23s

This is actually interesting cause there is not much opt done yet, it actually should be worst as it enumerates 40 versions iirc upfront.

[elcritch]

@Araq I remember starting out with Atlas when it was really early, and I thought the design was great. First, it created, one single root workspace folder where all the packages were installed. So it was just a giant folder with all the packages, and they were all being git checked out, and they were all ready for development. And I actually used that system for a while, and then I transitioned to using it at Reddit as well. And it was great, it was just install, and everything just made sense. But then, stuff stopped working. I guess it moved into where the Atlas now creates a deps folder into every package that you install, potentially creating multiple copies of all the deps. Maybe like npm?

Same for me, it was great. It was actually the move to SAT that broke old Atlas. The move to links was added later and had to fix all the SAT stuff first. :)

It's just not the way I develop at all. Yes, you can install them for development through these links, but then you have links everywhere. I didn't get it. Like, it was much simpler before, just a big folder and just a single nim.cfg, and that was it. But now you have a folder, and there's deps, and there's links, and there is pins, and there's other stuff.

Yeah links are so-so but they can be nice.

You can still go back to the shared-folder style Atlas by creating a atlas.config in your projects and setting {"deps": "../"} or whatever dir you want. Hmmm, I'll add some docs for that in Atlas. It's not exactly the same as the old system but pretty similar.

Like this:

-> % ls                     
AGENTS.md      docs           fidget2.nimble nim.cfg        src
atlas.config   examples       LICENSE        README.md      tests

-> % cat atlas.config 
{
  "deps": "../",
  "nameOverrides": {},
  "urlOverrides": {},
  "pkgOverrides": {},
  "plugins": "",
  "resolver": "SemVer",
  "graph": null
}                                                                                                         

-> % ls ../
_nimbles         bumpy            fidget2          nimsimd          sigils           winim
_packages        chroma           flatty           opengl           urlly            ws
atlas.cache.json cligen           jsony            pixie            vmath            x11
bitty            crunchy          libcurl          puppy            webby            zippy
boxy             deps             mddoc            shady            windy

It took me like 20 min of trial an error to get it to install fidget2 when you asked ... this is not the experience I had when I first started with it.

Odd, did you fix anything in your Nimble? An atlas install just worked on a fresh clone of fidget2 for me.

[treeform]

It took me 20 min to figure out i needed to run atlas init because atlas install fidget2 and atlas install fidget2.nimble sort of worked without it but did not compile. I wish docs had "atlas use in CI" section.

[elcritch]

It took me 20 min to figure out i needed to run atlas init because atlas install fidget2 and atlas install fidget2.nimble sort of worked without it but did not compile.

Thanks for the feedback, I noticed the readme.md was pretty out of date after reading your comment and made a PR to fix it. Feel free to file issues for Atlas as well. I've been enjoying using Atlas and try to fixup bugs/issues when they come up. Luckily there's been relatively few.

I wish docs had "atlas use in CI" section.

Good idea! I use Atlas with CI and a Github Cache which makes it pretty fast.

I also added a section on using Atlas in a workspace style setup.

[elcritch]

The reason Nimble is slow seems to be due to how its pkgs2 folder is designed and works.

It does multiple git clones of a package for different versions and then copies the "path" to the pkgs2/. Or seemed to do that when I last looked into it.

Essentially Nimble is currently acting as a package manager where you download different versions directly but (has?) to do so by cloning the entire git repo.

Also copying the "paths" versions will occur much more overhead as well. Atlas had a speedup when it switched to using git file commands to get just the nimble files. The nimble files are then cached so it's pretty quick to enumerate and parse the entire dependency graph.

[jmgomez]

There are some optimisations in place now in the latest head. To fully activate them --asyncdownloads must be enabled. It will be on by default, but there are some issues to be fixed first.

nimble install in the fidget2 repo (clean install rm -rf ~/.nimble nimbledeps) Prev released nimble: nimble_0.20 install 104.78s user 63.46s system 50% cpu 5:36.25 total

nimble install --asyncdownloads 31.23s user 16.25s system 33% cpu 2:22.02 total

maximumTaggedVersions speed it a bit too, but its marginal.

More opts will be added before the final release. Let me know if you give it a shot and its consistent with your results