zef install ignores dependencies in some cases

[donpdonp]

After adding a new module to the depends: [] list, zef install . does not install the newly listed module.

Context

If the primary module in META6.json is installed, the deps are ignored. zef install . and zef install . --depsonly both say everything is installed when DBIish is not installed. Uninstalling the primary module, then running zef install . will install the newly listed module

## Expected Behavior
<!--- Tell us what should happen -->
I expect DBIish to be installed

## Actual Behavior
<!--- Tell us what happens instead -->
no modules are installed

## Steps to Reproduce
<!--- Provide a unambiguous set of steps to reproduce this bug. -->
<!--- Please use the --debug flag: e.g. `zef --debug ...` -->
$ zef -i list | grep DBIish => not found (expected)
Edit META6.json: "depends"     : ["Cro::HTTP", "URI", "DBIish"]
$ zef install --depsonly .
All candidates are currently installed

## Your Environment
<!--- Show the output from the following commands to tell us more about your environment -->
* perl6 -v
* zef list --installed
 $ perl6 -v
This is Rakudo version 2018.06 built on MoarVM version 2018.06

zef version is `zef:ver<0.4.5>:auth<github:ugexe>`

[ugexe]

This is by design since a given namespace/distribution is considered immutable. You could otherwise change any one of the name/auth/api/version.

[ugexe]

Alternatively:

zef install . --force-install --deps-only

[zoffixznet]

FWIW just stumbled over it myself, while debugging a supposedly failing installation of a module.

I uninstalled all the deps and I was expecting --depsonly to just check the dependencies and install any that are missing, without caring whether the module itself is already installed, but it kept telling me everything is already installed and I thought it was talking about the dependencies:

$ zef uninstall HTTP::UserAgent JSON::Fast IO::Socket::SSL
!!!> Found no matching candidates to uninstall

$ zef --depsonly install .
The following local path candidates are already installed: .
All candidates are currently installed

[sjn]

I also bumped into this now. My expectation was exactly the same as @zoffixznet's. Would it be meaningful to change the behavior when using --deps-only to actually check the dependencies - or at least check if the dependencies are met? Right now "zef --deps-only install ." doesn't even checking if the dependencies have changed, it seems.

The design decision that a given namespace is immutable should perhaps not be true when working with the "." dir? When installing "." I think it should be assumed one is a developer in the process of changing what is in the current directory...

[ugexe]

That disregards the fact that Rakudo itself won’t recognize any differences that zef would with the suggested changes. People would run into weird runtime issues which will essentially be impossible for me to debug unlike the much easier to debug missing dependencies issue. It easy to look at this from the perspective of “just do foo” but that just creates a bigger problem. It isn’t about zef not seeing something... it’s about working in a manner consistent with Rakudo to avoid problems after installation is over.

[niner]

Sounds to me like the proper solution to developer versions that works within rakudo's framework would involve some automatically generated version number, just like git describe does really. It may even simply involve git describe. That would get us from "you need to increase the version in META6.json every time you try to install" to "you need to commit stuff before installing".

Taking it one step further and learning from rpm, we could even add an installation counter (analog to rpm's build counter that gets added to the version), so you wouldn't even have to commit your changes.

Taken together the generated version would look like: <version from META6.json>.<git describe output>.<installation counter>

We would get that installation counter by looking up the highest installed version of <version from META6.json>.<git describe output>.* and increment the last part.

This just leaves the question: how do we know that we want to auto-generate versions in the first place?

[sjn]

That disregards the fact that Rakudo itself won’t recognize any differences that zef would with the suggested changes.

Are we talking about specific types of changes here, or of any changes? Because if I add a new dependency to the META6.json file, it means I actually need it installed for my software to be able to run. Rakudo already recognizes if a dependency is missing and gives an appropriate error message. What I'd like to do is to solve that error by installing the missing dependency, but as things are now, zef just says "it's already there" ("All candidates are currently installed") when that in fact is NOT the case! To me this is an actual bug in the behaviour of zef (or whatever other module is used to determine if something is already installed).

Alternatively, how about zef storing a SHA256 of META6.json in the same directory to keep an idea of what was actually installed the last time zef was run? If the checksums don't match, then zef knows that the metadata for this module has changed (which also implies we're working within a developer context), and can act accordingly...

People would run into weird runtime issues which will essentially be impossible for me to debug unlike the much easier to debug missing dependencies issue.

I'm not sure what kind of runtime issues you're talking about here - would you mind elaborating?

It easy to look at this from the perspective of “just do foo” but that just creates a bigger problem.

Not sure of what bigger problem you are talking about here, but whatever it is, it happens in a developer context. As I said, I'm unsure what kind of bigger problems you're talking about, but if I may guess, does it have to do with zef and Rakudo's abilities to distinguish between a dev environment and a production environment? (please correct me if my guess is off the mark)

It isn’t about zef not seeing something... it’s about working in a manner consistent with Rakudo to avoid problems after installation is over.

I'd love to learn more about this, if you don't mind. :-)

[niner]

There are 2 related issues being discussed in this ticket:

• zef install refusing to do install by default if the distro (version) is already installed
• zef install --deps-only refusing to do anything if the distro (version) is already installed

Despite the close relationship and one being just a special case of the other, it seems useful to differentiate between them, as our options and the results differ widely.

zef install --deps-only despite being just a special case of zef install can also be seen completely differently as "only take the dependency information of the source distro as a list of things to install". In the latter perspective, it does not make sense to even look if the distro is already installed as we are not interested in the distro itself. Also going on with installation will not cause any harm, as without installation of the distro, it will not conflict with the already installed one.

In the general case of zef install we must be cautious as @ugexe says and for that my suggestion above may be a possible solution.

[sjn]

There are 2 related issues being discussed in this ticket:

• zef install refusing to do install by default if the distro (version) is already installed
• zef install --deps-only refusing to do anything if the distro (version) is already installed

Despite the close relationship and one being just a special case of the other, it seems useful to differentiate between them, as our options and the results differ widely.

zef install --deps-only despite being just a special case of zef install can also be seen completely differently as "only take the dependency information of the source distro as a list of things to install". In the latter perspective, it does not make sense to even look if the distro is already installed as we are not interested in the distro itself. Also going on with installation will not cause any harm, as without installation of the distro, it will not conflict with the already installed one.

Yep, this sounds good to me, and this is the case I'm really concerned about. If zef's behaviour would be changed to something like what you and @zoffixznet describe, then this bug would be resolved as far as I'm concerned.

In the general case of zef install we must be cautious as @ugexe says and for that my suggestion above may be a possible solution.

Fair enough. Still unsure what specific scenarios could create problems, but if this is true, maybe it deserves a separate issue? I guess some sanity checks could be in order? Or a separate installation destination for files that are being actively developed at the moment?

[niner]

I guess some sanity checks could be in order?

Those are already there - hence the "All candidates are currently installed" and refusal to install.

Or a separate installation destination for files that are being actively developed at the moment?

There is - the FileSystem repository you get by simply adding -I. to your raku command.

[sjn]

Those are already there - hence the "All candidates are currently installed" and refusal to install.

I was refering to adding sanity checks to avoid the risky installation scenarios the two of you are concerned about. If these potentially-breaking installation scenarios already have sanity checks (and consequential refusal to install), then what are you worried about?

There is - the FileSystem repository you get by simply adding -I. to your raku command.

I'm suggesting making zef install somewhere "dev-specific" without having to be explicit about it (and perhaps have this mirrored by Raku so things work in dev without "-I." ?) Dunno if that's a good idea though. :)

[ugexe]

The sanity checks are precisely what is being suggesting to be removed. What if you have a version of a dist already installed that expects certain versions and then you go and change those dependencies without changing your dist version? As a tool meant for production systems it should strive to maintain sanity of the system over making development easier.

Treating some paths special breaks consistency. Consistency is goal.

Using git describe goes against the generic nature of repositories. At that point it might as well go into a development tool anyway.

Installing somewhere dev specific that rakudo still sees would be introduce shadowing problems. Not good for production systems. Sounds like something for a Raku-env tool.

Even if the suggested change was made what should it do for transitive dependencies? I guess it should special case that too since it would not normally do that? Again, this breaks consistency.

Ultimately this behavior can not be shoe horned into ‘zef install —deps-only’ without being inconsistent. I guess a new command works around this technically, but I not sure how I feel about adding a development-helper sub command. Especially since running with —debug makes it clear it’s not even looking for the dependencies (which isn’t so clear without —debug).