Consider removing the workspace vs project distinction; bring back .nimble-links

[Araq]

Refs: https://github.com/nim-lang/atlas/issues/104

I grow more and more unhappy with Atlas's solution to how it manages things and its complexity. The distinction between project and workspace is annoying in practice as I typically have a single "projects" directory that should not be Atlas's workspace as there are too many unrelated projects in it.

The problem Atlas tries to solve is easy development of multiple projects at the same time. This does not necessarily require the notion of a workspace. Symlinks would also work but since these are not portable we can use Nimble's idea of using .nimble-link files.

In other words Atlas manages a single project that has a vendor/ or deps/ directory where its dependencies are stored. Either directly inside this directory or by a linkfile that links to where to find the project. This means the dependency graph always has a single root and thus the command line interface is simplified too. It is always clear what a "lockfile" is. It is also always clear where the nim.cfg goes and how it should be maintained.

[Clonkk]

The way I ended-up using Atlas :

Workspace/
  atlas.workspace
  nim.cfg # Contains path to deps/ and src/
  deps/atlas.lock # contains external deps
  projects/ # This is a submodule w/ private code that depends on deps. Libraries only, no binary
    config.nims 
    lib1/
    lib2/
  main/ # contains nim file compiled into binary
    main_service1.nim
    main_service2.nim      
  # non-nim stuff
  scripts/ # collections of bash scrips to deploy stuff (.rpm, .deb packager mostly)
  cfg/ # default config files to deploy when installing

• • Inside projects were a collection of private Nim libraries (they were submodules so not nimble package). In retrospect, it should have been a single submodule and not 1 repo / library.
  • • The only reason I wanted a submodule was to keep track of the commit. This could be solved by having multiple atlas.lock file or a global one.
• • I used multiple, different workspace if I wanted to have variations e.g. :

Workspace/
  atlas.workspace
  nim.cfg # Contains path to deps/ and src/
  deps/atlas.lock # contains external deps
  projects/ # Submodule that can be a different commit hash or a different project entirely
    config.nims
    lib3/
    lib4/
      
  main/
    main_service3.nim
    main_service4.nim
    
  scripts/ # collections of bash scrips to generate installer (.rpb or .deb)
  cfg/ # default config files to deploy when installing   

So essentially :

• I never used "multiple independent project per workspace" and used multiple workspace instead.
  • I used atlas workspace as a template to deploy services / binary (whatever)
• I needed to differentiate private code from external code in a clear manner.
• A pain point were external deps/ that were dependent on C library / shared object that had a global install version.
  • See https://github.com/nim-lang/nimble/issues/918 which is similar in nature
• I used atlas env in the workspace to fix the compiler version (it was a bit annoying having to recompile each time so sometimes I would 'cheat' by using symlink to a global install between my workspace)

[jfilby]

What about a project tree? That way you could have a single project, or multiple projects organized in a tree.

[moigagoo]

100% agree with the proposal. I tried and failed to use Atlas multiple times specifically because I it made me do the dance with my existing folder structure, which is, I assume, and more or less standard one:

1. ~/Projects is where all my projects live.
2. ~/Projects/projectname is there my project's nimble file sits.
3. For multiservice projects it's ~/Projects/projectname/[service1, service2, etc.], each service dir with its own nimble file (if it's a Nim project).

[lou15b]

I am hoping that Atlas will expand the concept of "project" to allow the development of several related Nim packages at the same time. As far as I can understand, I have not yet seen this capability.

For example, one may be developing a UI package, which uses opengl and sdl2. The project may include:

• the UI package under development
• a new package of opengl helpers, which is being developed
• a new package of sdl2 helpers, also being developed
• an enhanced version of some existing package, but the enhancements have not yet been solidified, so a PR has not yet been submitted for the package
• ... and so on.

The project requires all of the above development packages to use the same dependencies directory, which contains existing packages such as opengl and sdl2, and also pointers to the packages under development.

What is missing is the ability to automatically generate links in the dependencies directory, pointing to the packages under development.

This is a real need, at least for me. And I expect that others developing projects of a significant size also have the same need.

Or have I missed something, and the capability already exists?

[Araq]

Or have I missed something, and the capability already exists?

There just isn't a need for it at all.

Instead of "atlas, use path/to/package/in/development" you would simply patch the nim.cfg with --path:path/to/package/in/development.

[elcritch]

Definitely seems promising. The workspace vs project distinction adds complexity which also makes getting all the paths correct a pain. It complicates a lot of the code.

Though, how would you link projectB to use projectA's dependencies? It reminds me a bit of pyenv where you create an environment and can use it across multiple projects.

[elcritch]

@Clonkk @lou15b the new link feature just landed. It removes workspaces. If y'all would like to test it out I'll keep an eye out on features.

[Araq]

Has been implemented.

[Clonkk]

I'll test it out