Clarify current fragility of `test/Project.toml` approach

[cscherrer]

@fredrikekre 's comment here says,

Don't use the functionality of test/Project.tomls which is pretty beta (and documented as such)

But this is not clear at all from current documentation here, which only says

The exact interaction between Project.toml, test/Project.toml and their corresponding Manifest.tomls are not fully worked out, and may be subject to change in future versions. The old method of adding test-specific dependencies, described in the next section, will therefore be supported throughout all Julia 1.X releases.

The section title here is "Test-specific dependencies in Julia 1.2 and above", and the qualification seems to me to only indicate that the current approach may change in the future, not that it's "beta" in any way.

I have set up a few repositories using this approach, only to find out today that it's extremely brittle. Can the current documentation be updated to clarify this?

[phipsgabler]

There's something else that's not entirely clear here -- the docs say that "Pkg will add the tested package itself implictly". This works fine when running ] test, but I also expected the following to behave the same (that's on Julia 1.3.1):

(AutoGibbs) pkg> activate test
Activating environment at `~/.julia/dev/AutoGibbs/test/Project.toml`

julia> using AutoGibbs
ERROR: ArgumentError: Package AutoGibbs not found in current path:
- Run `import Pkg; Pkg.add("AutoGibbs")` to install the AutoGibbs package.

I guess this is related to the fact that the sub-project thing is still very much under discussion?

[KristofferC]

I guess this is related to the fact that the sub-project thing is still very much under discussion?

Yes, this would work with subprojects but with the current design, Pkg.test does "magic" to put these things into a new project that includes everything. My goal is to get subprojects into 1.6 (which will likely be the new LTS).

[FelixBenning]

I just tried this approach and I can not get it to work:

test/Project.toml

[deps]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40

if I activate . :

(MyProject) pkg> test
....
ERROR: LoadError: ArgumentError: Package Tests not found in current path:
- Run `import Pkg; Pkg.add("Tests")` to install the Tests package.

if I activate ./test I can neither do

(test) pkg> test
ERROR: trying to test unnamed project

nor

(test) pkg>  test MyPackage
ERROR: The following package names could not be resolved:
 * MyPackage (not found in project or manifest)

I also tried adding the uuid of MyPackage to the test/Project.toml which complains about the package not being registered (of course).

[fonsp]

@FelixBenning from your error (Package Tests not found in current path) it looks like you did using Tests instead of using Test. (Which is an understandable mistake! 🙃)

[cossio]

Are there any updates on which direction test dependencies (and subprojects generally, such as docs) will move? Should we continue to use test/Project.toml?

[acdupont]

test/Project.toml helps with startup time of the main project because test specific dependencies don't have to be in the main Project.toml. I hope that test/Project.toml is continued to be supported.

[dpinol]

We find this feature very useful as we don't only have test/Project.toml, but also test-performance/Project.toml and others

[IanButterworth]

@acdupont That's not the case. Test dependencies in Project.toml listed in the [extras] section do not get installed or precompiled during Pkg.add and obviously not loaded.

[acdupont]

@acdupont That's not the case. Test dependencies in Project.toml listed in the [extras] section do not get installed or precompiled during Pkg.add and obviously not loaded.

What do you mean by "That's not the case"? You proposed an alternate way for sub-project dependencies to be ignored by the main project, but using separate Project.toml would work for this as well. I prefer sub-project dependencies to not be listed in the main Project.toml at all.

[IanButterworth]

test/Project.toml helps with startup time of the main project because test specific dependencies don't have to be in the main Project.toml

With Project.toml test-only (and build-only) deps can be listed in an [extras] section, with a target for test assigning them as test-only deps.

What you wrote gives the impression that the only way to do that is via test/Project.toml, which is not true.