dub icon indicating copy to clipboard operation
dub copied to clipboard

On-demand build of imported source files

Open nordlow opened this issue 4 years ago • 17 comments
trafficstars

dub finds all *.d files and passes them to the compiler, whether any other code uses that module or not. This is in contrast to dmd -i, which only causes the compiler to look at a D source file if it's actually imported.

There might be cases where dub could benefit from such a behavior via a flag. At least when building applications,

nordlow avatar Oct 20 '21 11:10 nordlow

should probably only do this for executable ~~and unittest~~ targets

WebFreak001 avatar Oct 20 '21 13:10 WebFreak001

Definitely not for unittests. We have modules that are just unittest modules and aren't imported by anyone.

Geod24 avatar Oct 20 '21 14:10 Geod24

What about files with static this ctors? Executables can rely on modules with those and as far as I'm understanding, this will make those modules not behave as intended.

ljmf00 avatar Oct 21 '21 01:10 ljmf00

That's also a concern. Although this style of coding is not prevalent AFAIK. I think we should sample the existing ecosystem to see what people need / go to naturally.

In my case, we introduced the mainSourceFile change (that has drawn some controversy) because our main project is:

  • An application (server);
  • With some tools (a config file validator, a small client);
  • Has a complex unittest structure (we have some "tools" that are generated to check our C++ bindings);
  • And is used as a library by other programs;

This we solved with:

  • Having the server configuration first and having some excludes (https://github.com/bosagora/agora/blob/ab158eadaf30aa6bb8e335a09921365af909cb98/dub.json#L41-L55)
  • Having a ridiculously long exclusion list for the client (https://github.com/bosagora/agora/blob/ab158eadaf30aa6bb8e335a09921365af909cb98/dub.json#L56-L83)
  • Using targetType: none for the "library": https://github.com/bosagora/agora/blob/ab158eadaf30aa6bb8e335a09921365af909cb98/dub.json#L102-L105

The last two solutions are hacks. The targetType, especially, forces us to essentially bypass dub when using it as a library.

So I fully support having a way to have the -i behavior. I'm wondering if, after a transition period, it should even be made the default. But as mentioned, we also have unittest-only modules which no one import.

Geod24 avatar Oct 21 '21 02:10 Geod24

aren't static this ctors only called when their module is imported and something from the module is used in some way?

WebFreak001 avatar Oct 21 '21 05:10 WebFreak001

aren't static this ctors only called when their module is imported and something from the module is used in some way?

According to https://dlang.org/spec/module.html#staticorder they run are run regardless of being imported or not.

nordlow avatar Oct 21 '21 08:10 nordlow

Given that this has the potential to break people's code (i.e. self-registering routes), has anyone run a profiler on dub to see where it actually spends its time as part of slowing down compilation?

rikkimax avatar Oct 22 '21 23:10 rikkimax

Given that this has the potential to break people's code (i.e. self-registering routes), has anyone run a profiler on dub to see where it actually spends its time as part of slowing down compilation?

This can benefit from huge libraries like vibe-d. If someone only uses 2 or 3 files and they are statically linked, there is no need to compile the whole thing as the unused symbols will get discarded anyway.

aren't static this ctors only called when their module is imported and something from the module is used in some way?

According to dlang.org/spec/module.html#staticorder they run are run regardless of being imported or not.

I'm not completely sure if that is the current behaviour, although.

ljmf00 avatar Oct 23 '21 02:10 ljmf00

possible feature for dud?

WebFreak001 avatar Oct 26 '21 09:10 WebFreak001

possible feature for dud?

Ping, @burner.

nordlow avatar Oct 26 '21 10:10 nordlow

I'm not quite sure if I really understand the purpose/benefit.

Could somebody please add an issue to dud's github https://github.com/symmetryinvestments/dud linking back to this one here

burner avatar Oct 27 '21 07:10 burner

Could somebody please add an issue to dud's github https://github.com/symmetryinvestments/dud linking back to this one here

dmd -i <set_of_initial_d_files> only compiles the files that set_of_initial_d_files actually imports. This cannot currently be expressed by dub. I'm not sure if such a behaviour would be relevant to have in dud.

nordlow avatar Oct 27 '21 08:10 nordlow

hm, you can exclude and include specific files in the dub config can't you

burner avatar Oct 27 '21 08:10 burner

You can but it's a pain, if you want to have multiple binaries in the same package. Using subpackages is a pain, so in my experience people tend to reach for configurations instead. Problem is, dub will compile everything in source, so if you add a new source file, you have to exclude it in all your configurations. Not really scalable. See my comment above for an example of such usage.

Geod24 avatar Oct 27 '21 08:10 Geod24

playing devils advocate here.

sounds like you want variables and basic set operations to create an array of the files you want to exclude and then say something like sourceFiles / YOU_VARIABLE HERE

burner avatar Oct 27 '21 08:10 burner

I want recursive exclusion, so I can say "exclude source/myAppName/". Currently I need to say "exclude source/myAppName/* but also source/myAppName/*/*", etc... But even this has issues, because of when globbing is performed. See for example https://github.com/dlang/dub/issues/2142

Although, even if exclusion / inclusion was smarter (it should be), excluding the same pattern for every other configuration would still be annoying boilerplate. Acceptable boilerplate, but boilerplate nonetheless.

Geod24 avatar Oct 27 '21 09:10 Geod24

Looking at all the special things I have come across in regards to build file configuration languages an approach similar to what reggae does makes sense, aka. write a D program that creates the actual "makefile".

Making this api a first class citizen in dud is likely the better idea. In the sense that when

  1. dud init does no longer work
  2. you edit du(b|d).json as currently defined
  3. if that fails you write a program that calls libdud

burner avatar Oct 27 '21 09:10 burner