Absolute paths in `package.mill` hinder isolated compilation of nested submodules

[Yan-Muzi]

The package declaration requirements introduced in Mill 0.12.x (#3426) create conflicts in projects with nested submodules that need to function both as project dependencies and standalone components.

Take XiangShan as an example. We have a submodule rocket-chip, which contains cde as its submodule. When cde is used in XianShan, cde directory requires package build.rocket-chip.cde. However, when we want to develop rocket-chip independently, the same cde directory now requires package build.cde.

Is there any ways to fix or bypass this?

[lefou]

I was thinking about making the top-level package configurable yesterday, e.g. for cases where build doesn't match (e.g. you want to import some library which also has top-level package build). It's not there, but it might be suitable to fix your issue as well. But at the price, that the independent project needs to use the long package as well

//| mill-root-package: build.rocket-chip

[lihaoyi]

@Yan-Muzi we don't have any first-class support for the scenario you describe. We could probably come up with something e.g. what @lefou discussed.

[Yan-Muzi]

Am I correct in understanding that I may configure the top-level package as either XiangShan or rocket-chip within cde? This sounds OK but is rather inelegant as a submodule should not be aware of its containing directory.

If relative paths are not acceptable, is it possible to define all the modules in the root directory?

[lihaoyi]

@Yan-Muzi it is possible to define all the modules in the root directory build.mill. Could you explain a bit how that would help?

[Yan-Muzi]

If all the modules can be defined in the root directory, then it is the root directory that worries about its submodules instead of otherwise. This seems better because a git repository has all the information of its submodules, while it is unaware of how it will be used in other repositories as submodule.

For example, in a repository with nested submodules like this:

├── build.mill
└── submodule-A/
    └── build.mill
    └── subsubmodule-B/
        └── build.mill

build.mill in root directory can describe how submodule A and subsubmodule B are be used in the project, and build.mill in submodule A can describe how it uses subsubmodule B, which are different as the former is package build.submodule-A.subsubmodule-B while the latter is package build.subsubmodule-B. In this way, submodule-A can compile alone without the information of root directory.

[lihaoyi]

@Yan-Muzi got it. I think we may have some sanity-check assertions that yell at you if you have a build.mill in a sub-folder, because it's a pretty easy mistake to make when people should create a package.mill folder instead. But maybe we can offer some way to silence that error

[Yan-Muzi]

Looking back on #3426, I found this:

We can look into making package declarations optional again in future, but for now this gives us working IDE support with minimal effort, converging with vanilla Scala, at the cost of a little boilerplate per-file.

Optional package declarations should solve this issue elegantly! Looking forward to this new feature.

[KireinaHoro]

Would like to give this a bump since right now our stuff are stuck on Mill 0.11.x due to this.

I've been thinking about how to repurpose the multi-file builds support and ask the "dual use" projects (e.g. rocket-chip in @Yan-Muzi's example) to include most of its build logic as traits inside package.mill, while exporting the necessary module (object)s inside build.mill when they are invoked directly.

For your reference, this is how the submodule use case used to work; take SpinalHDL as an example, the build logic for the library is first defined as a trait (Lib) and then exported locally as an object (lib) for tasks like publishing to Maven:

trait SpinalModule extends SbtModule with CrossSbtModule { ... }
object lib extends Cross[Lib](Version.SpinalVersion.compilers){
  def defaultCrossSegments = Seq(Version.SpinalVersion.compilers.head)
}
trait Lib extends SpinalModule with SpinalPublishModule {
  def mainClass = Some("spinal.lib")
  def moduleDeps = Seq(coreMod(), simMod())
  def scalacOptions = super.scalacOptions() ++ idslpluginMod().pluginOptions()
  def ivyDeps = super.ivyDeps() ++ Agg(ivy"commons-io:commons-io:2.11.0", ivy"org.scalatest::scalatest:${scalatestVersion}",
    ivy"io.github.zhaokunhu::ipxactscalacases:0.0.3")
  def publishVersion = Version.SpinalVersion.lib
}

A downstream dependency would include the project as a submodule, use a $file-based import, and then extend Lib with the necessary path overrides:

import $file.deps.spinalhdl.build
trait MySpinal { this: deps.spinalhdl.build.SpinalModule =>
  def name: String
  def crossValue = v.scalaVersion
  override def millSourcePath = os.pwd / "deps" / "spinalhdl" / name

  override def coreMod = ModuleRef(spinalCore)
  override def libMod = ModuleRef(spinalLib)
  override def idslpluginMod = ModuleRef(spinalIdslPlugin)
}
object spinalLib extends deps.spinalhdl.build.Lib with MySpinal { def name = "lib" }

---

With multi-file builds and the new package.mill and build.mill split, I see the potential of using the same paradigm. In SpinalHDL, trait Lib can be defined inside package.mill:

package build // is this ok?

trait Lib { ... }

while object lib should be defined inside build.mill:

package build

object lib { ... }

Then, in the downstream project, assuming the SpinalHDL project is a submodule at deps/spinalhdl:

package build

trait MySpinal { this: deps.spinalhdl.SpinalModule => ...}
object spinalLib extends deps.spinalhdl.Lib with MySpinal { ... }

(with the empty placeholder package.mill inside deps/, same as the docs describe)

The problem I see here, which is the same problem raised in this issue, is that there needs to be some kind of subpath replacement happening, effectively replacing package build for the submodule subtree to package build.deps.spinalhdl. Would this be possible by Mill?

Thanks very much for your time.

[lihaoyi]

@KireinaHoro currently, the assertion that the package name matches the folder structure is here:

https://github.com/com-lihaoyi/mill/blob/f2d39e8e75f642de9b1e94cd93ff6f5082398d0f/runner/meta/src/mill/meta/FileImportGraph.scala#L77-L84

And the code that assumes that nested build files are named package.mill lies here:

https://github.com/com-lihaoyi/mill/blob/f2d39e8e75f642de9b1e94cd93ff6f5082398d0f/runner/meta/src/mill/meta/FileImportGraph.scala#L144

Both of these can be easily changed: we could allow nested build files to be named build.mill, and we could loosen the restriction on the package name matching the folder structure and just rewrite it to the correct package statement.

If you're interested it should be a relatively easy change to make, and I think it should solve your issue and let you directly reference the build.mill files of subprojects within the parent project. Maybe we can hide it under a flag so you can try it out and find any issues before opening it up for others to use

[KireinaHoro]

@lihaoyi thanks! That sounds very good, I'd be up to try it out. One thing I haven't quite figured out yet reading your suggestions, is that how would the parent project refer to the subprojects' types: would the fully qualified name be build.<path>.<to>.<build.mill>.MyType (implying some kind of "scoped/prefixed import")? Or would it just flatten out all types into the build package in the parent?

[lihaoyi]

One thing I haven't quite figured out yet reading your suggestions, is that how would the parent project refer to the subprojects' types: would the fully qualified name be build...<build.mill>.MyType (implying some kind of "scoped/prefixed import")? Or would it just flatten out all types into the build package in the parent?

@KireinaHoro For the minimal possible change, we'd just treat the nested build.mill files as if they were package.mill files in the same folder. That would mean you would access them via build.<path>.<to>.<build.mill>.MyType

There's developer documentation on the Mill readme, so feel free to take a crack at making these changes and trying them out on your own small demo projects to see how it goes. If it looks like it does what you need, you can then open a PR and we can get it merged into mainline so you guys can upgrade

[KireinaHoro]

For the minimal possible change, we'd just treat the nested build.mill files as if they were package.mill files in the same folder. That would mean you would access them via build.<path>.<to>.<build.mill>.MyType

@lihaoyi ok I see how that would work for the consumer side. However, the package build clause declared within the subproject's build.mill will not match with the "actual path", which is <path>/<to>/<build.mill>/build.mill. I recall that (at least in normal Scala) the package clause is the authoritative path declaration, not the directory structure? This is why I had the impression that there will just be a "flattened namespace" i.e. MyType would be build.MyType, instead of under build.<path>.<to>.<build.mill>.MyType. Is Mill doing anything special under the hood regarding the directory structure?

[lihaoyi]

@KireinaHoro Mill over-writes the package statement here (https://github.com/com-lihaoyi/mill/blob/f2d39e8e75f642de9b1e94cd93ff6f5082398d0f/runner/meta/src/mill/meta/CodeGen.scala#L122), so even if the package statement in the source code is wrong, if you turn off the error as I explained above, I think it should just work (although IDEs like intellij may be confused)

[KireinaHoro]

Thanks! I'll give it a crack and see if it would work the way we need.