[RFC] Cyclic imports and symbol dependencies

[yglukhov]

trafficstars

This is a feature request to allow cyclic imports. This allows to define mutually dependent types and procs (templates/macros) in different modules or in the same module regardless their definition order. Example:

# bar.nim
import foo

type Bar* = ref object
    f*: Foo

proc doSmthWithBar*(b: Bar) = discard

when isMainModule:
    let f = Foo.new()
    let b = Bar.new()
    f.doSmthWithFoo()
    f.b.doSmthWithBar()
    b.doSmthWithBar()
    b.f.doSmthWithFoo()

# foo.nim
import bar

type Foo* = ref object
    b*: Bar

proc doSmthWithFoo*(f: Foo) = discard

when isMainModule:
    let f = Foo.new()
    let b = Bar.new()
    f.doSmthWithFoo()
    f.b.doSmthWithBar()
    b.doSmthWithBar()
    b.f.doSmthWithFoo()

The symbol may be subjected to a cyclic lookup only if the following conditions are met:

• The symbol is a top level symbol.
• The symbol is defined by hand and is not a result of macro/template evaluation.
• The type of the symbol is not dependent on another template/macro global evaluation.

Forum discussion: http://forum.nim-lang.org/t/2114

[endragor]

• The symbol is defined by hand and is not a result of macro/template evaluation.
• The body of the symbol is not dependent on some template/macro evaluation.

Is it possible to ease these limitations somehow?

Why does body content matter here? I think large portion of Nim procedures are dependent on template evaluation - there are lots of them in stdlib and Nim promotes usage of templates to reduce boilerplate code.

[timotheecour]

note: there is a (very) partial support for cyclic imports, see http://nim-lang.github.io/Nim/manual.html#modules

Recursive module dependencies are allowed, but slightly subtle

(see corresponding example + limitations)

[Zireael07]

I just ran into a similar limitation, where one type refers to another, sort of a child-parent relationship. Can this get fixed soonish? It's been open for several years...

[liquidev]

Any progress on this? It's really a pain when doing game dev, you have to split your modules in weird ways to achieve what you want.

[chr-1x]

At the risk of making a post that basically boils down to "+1", this has been probably the biggest pain point for me with Nim so far.

To add a little more substance, here's something that's trivial to do in C/C++ and java-like languages (not to even mention scripting languages), but awkward to do in Nim:

widget.nim:

import processor

type
  Widget* = object
    processorOption: ProcessorOption

processor.nim:

import widget

type
  ProcessorOption* = enum
    ProcessFast
    ProcessSlow

proc process*(widget: Widget) =
  internalProcess(widget.processorOption)

Today, this yields the following error:

/cyclicnim/widget.nim(5, 22) Error: undeclared identifier: 'ProcessorOption'
This might be caused by a recursive module dependency:
/cyclicnim/processor.nim imports /cyclicnim/widget.nim
/cyclicnim/widget.nim imports /cyclicnim/processor.nim

What are the refactor options for fixing this error?

• Move processorOption out of processor.nim into a third module: Awkward, because processorOption is clearly related to processing things, and users of processor will not appreciate having to add two imports to get the whole functionality of processor.
• Move process(Widget) out of processor.nim into widget.nim: Awkward, because if we have the entry point for actual processing outside of processor.nim, what's it even for!? Besides, it calls internalProcess, which we clearly don't want to make public.
• Move import widget in processor.nim below ProcessorOption declaration: Awkward, because now not all our imports are at the top of the file. Also, this doesn't generalize well to larger modules with more inter-dependencies.
• Use include widget instead of import, with the {.experimental: codeReordering.} pragma: Awkward, because includes might not have the semantics we want (now we can access all the internal symbols!), widget doesn't have any visible reference to processor other than using its type, and now widget cannot be included on its own. Also produces this warning, despite compiling fine: Warning: Circular dependency detected. `codeReordering` pragma may not be able to reorder some nodes properely [User]
• Just smash the two modules together, and reorder things yourself: Awkward, because these are primarily dealing with different things, even though they happen to refer to each other.

Not a solution:

• Forward declare ProcessorOption in widget.nim: results in "implementation of 'ProcessorOption' expected". Making the enum value a ref, ptr, or seq is a non-option because they significantly change the semantics. This also applies if ProcessorOption was, say, an object.

How do other languages solve this problem?

• In C/C++, there are a few solutions, none of them are particularly nice but they are well understood and frequently used: move the #include, forward-declare the enum (requires c++11 enum class so the compiler knows the size), or move the enum to a new header. The latter is less objectionable than creating a new nim module because it doesn't imply anything about visibility or linkage, and #includes are the default in C++. In any case, I really hope Nim can do better than C++ here. Also, I don't know enough about the new modules TS to comment on how it will handle this problem.
• In C# (and maybe Java?), package-level namespacing and arbitrary-order declarations make this trivial. In C# you don't even have to explicitly import anything if they are declared in the same package. I would love if Nim had a package-style namespacing system like this.
• In Python, duck typing more or less obviates the issue. If using type annotations, you can use from __future__ import annotations as of Python 3.7 to defer type resolution until later (putting the onus on tools to resolve the cycle, but they probably have more complicated symbol resolving logic anyway). Also, modules can be organized into packages using __init__.py files, which allows users to have a single import.
• I'm less familiar with Javascript, but the ES6 modules appear to support such cyclic declarations without much effort on the part of the developer using explicit exports: https://stackoverflow.com/questions/46246383/cyclic-dependencies-in-javascript-modules-es6

[SolitudeSF]

Move processorOption out of processor.nim into a third module: Awkward, because processorOption is clearly related to processing things, and users of processor will not appreciate having to add two imports to get the whole functionality of processor.

you could just export the processorOption

[chr-1x]

Move processorOption out of processor.nim into a third module: Awkward, because processorOption is clearly related to processing things, and users of processor will not appreciate having to add two imports to get the whole functionality of processor.

you could just export the processorOption

I was lazy with the export markers (*), I should have pasted my entire test file where I verified the problems described in the post. Unless you're referring to a feature I'm not aware of?

[SolitudeSF]

i mean export processoroptionmodule from processor.nim so user doesnt have to import two modules

[krux02]

First of all, I would really like to improve the usability of Nim for multi module projects. But I don't think we can do a prototype prepass like you describe here, and here is why:

The problem I see is, a function signature is not just what is visible at the first line. Nim also has an effect system. Many effects are inferred automatically. These inferred effects are attached automatically, but they are necessary to know, to compute the effects of other procedures. This also applies to forward declarations. A forward declaration should list all effects of that function. If it does not do that, the compiler won't be able to inject them in a later compilation stage.

[krux02]

@chr-1x The way to deal with the current module system is to sort things by their dependency. This means if you have things that actually depend on each other, it is often a good idea to make it a single module. Put types that depend on each other in a single type section. But in your case, you don't have a cycling dependency, you can fix the cycle dependency by moving proc process from widget.nim to processor.nim.

[zah]

@Araq keeps claiming that my original attempt to solve the cyclic dependencies problem through the noforward pragma is not compatible with the effects system, but I've never seen a compelling evidence for this. My own understanding to this day is that the algorithm described in the documentation can solve all problems:

https://github.com/nim-lang/Nim/commit/1d29d24465ddb9aaea18558f221ff67bf82db0c7#diff-d86fb8f908ad34638ec054b961e99424R4651

The multi-module support with the noforward pragma has some challenges and the required refactoring in the compiler is quite substantial, but it's worth it IMO. It will bring some other benefits such as making nimsuggest significantly faster.

[al6x]

Having same problem, hard to structure project and I forced to create artificial and unneeded modules like a.nim, b.nim and shared_ab.nim. Instead of just a.nim and b.nim

Some may argue that circular module dependencies are bad, like they breaks levelled architecture and have other problems. But here the case is different. Those problems are if you have long distance circular dependencies between different layers etc. And here we are talking about short distance circular dependencies, that are frequently is just a more convenient way to re-arrange code in large module into smaller chunks as separate modules.

[Araq]

We don't want to implement a solution that accentically prevents incremental compilation from working so the priority was put on IC. Now that IC is slowly beginning to work, we are looking into how to support recursive modules.

[johnnovak]

Now that IC is slowly beginning to work, we are looking into how to support recursive modules.

Great new, this has been a constant source of frustration for me (and I'm guessing for many others).

[avahe-kellenberger]

Has any progress been made on this front now that IC has been in the works for a while? Curious about the progress, would love to have cyclic imports supported.

[arkanoid87]

I had to drop Nim on many projects due to this. Without a proper solution, many software design patterns become a bottleneck as soon the project reaches a certain size. Solution would be stick to procedural paradigm and strict idiomatic Nim, and say goodbye to flexibility and pattern mimicry when doing FFI

[zevv]

Stretch goal for 2023: https://github.com/nim-lang/RFCs/issues/503