embedded/bin should never be in the user's $PATH

[docwhat]

The /opt/chefdk/embedded/bin/ directory should never be in a user's $PATH. It contains executables that can interfere with with the normal working of a users system, such as clear, tput, xz, xml2-config, xslt-config, etc.

According to @lamont-granquist it was never intended to ever be put in the user's $PATH: chef/omnibus-chef#340

However, chef shell-init does add embedded/bin to the path:

$ chef shell-init bash
export PATH="/opt/chefdk/bin:/Users/docwhat/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/opt/chefdk/bin:..."
export GEM_ROOT="/opt/chefdk/embedded/lib/ruby/gems/2.1.0"
export GEM_HOME="/Users/docwhat/.chefdk/gem/ruby/2.1.0"
export GEM_PATH="/Users/docwhat/.chefdk/gem/ruby/2.1.0:/opt/chefdk/embedded/lib/ruby/gems/2.1.0"

Further, ChefDK is crippled if you don't add embedded/bin to the $PATH

[lamont-granquist]

"ChefDK is crippled" is overstating it. The problem is that rbenv and chruby users want all the ruby things to be colocated together in one directory to switch on (ruby, gem, rspec, chef, chef-client, knife, etc), but xml2-config and xslt-config and such are going to troll those users if they're put in the $PATH. With chef-dk since we want to expose 'ruby' then it should not be in embedded, or should not only be in embedded.

We still do not want symlinks from /usr/bin to ruby so it breaks our prior model of setting up symlinks from /usr/bin to everything in /opt/thing/bin, but it seems to me better to break that model.

We probably want ruby stuff in both places and want a symlink pointing one way or the other -- otherwise cleaning up and switching all the shebangs only for chefdk will be a nightmare. We still do want to be able to hit ruby in /opt/chefdk/embedded/bin.

[docwhat]

I dislike the symlinks in /use/bin. I'd rather everything was self contained for easy upgrading, uninstalling, and to prevent interfering with the OS files.

I said crippled because I expected knife, food critic, kitchen, etc. to work without Gemfiles. Which it does if you use chef shell-unit. I admit I'm not the inexperienced ruby/unix user ChefDK is trying to help, so maybe I'm thinking about it wrong.

I'm an rbenv user and have written a plugin that works around the embedded/bin booby-trap: docwhat/rbenv-chefdk

That's how I found these issues as I dug through the chefdk.

Anyone symlinking /opt/chefdk/embedded to $RBENV_ROOT/versions/chefdk is going to be in for a rude surprise. :grinning:

[danielsdeleo]

Okay, so what exactly is the proposal here? I think we'd at least want ruby and every default gem binary (e.g., the pry command`) to be in the PATH after a shell-init. I wouldn't mind if the rest of the stuff wasn't in the PATH by default, but I'd be wary about breaking user gem installs or other knock-on effects.

[docwhat]

1. Don't include embedded/bin in $PATH
2. Move all Gem binaries into bin/
3. Move embedded/lib/ruby to lib/ruby
4. Don't symlink anything other than the chef binary in /use/bin.

The reasoning is to make the top level directory structure as rubyesque as possible to ease new users into ruby. A new user can more easily look thing up on the Internet and see them in /opt/chefdk.

This will allow it to work with RVM and RBEnv and friends.

[lamont-granquist]

Yeah, so all the ruby things should be in /opt/chef/bin, that includes pry, erb, gem, bundle, etc -- anything in the ruby distro itself, and anything installed from a gem.

All of those should also probably be symlinked one way or the other into embedded/bin as well (although I don't think this is a hard requirement but otherwise we'd have to fix a lot of shebangs if /opt/chefdk/embedded/bin/ruby goes away).

Then things like xml2-config, xz and tput should be isolated into embedded/bin.

That way there's three buckets:

• stuff that gets symlinked into /usr/bin and tries to be "minimally invasive" to the distro and the distro ruby
• /opt/chef/bin which is for "put this in the path and use chefdk as my ruby instead of the system ruby"
• /opt/chef/embedded/bin which is for "chef-client may use this internally and add it to its own PATH, but nobody should be touching this directly

[docwhat]

Do you mean /opt/chefdk instead of /opt/chef. Just checking. I imagine that chef has similar problems to chefdk but usually you don't add chef to paths.

[lamont-granquist]

Yeah, for consistency we probably should apply this to chef as well. I don't think it would be a breaking change.

[johnbellone]

Any status here?

[lamont-granquist]

if we do this there's a bit of conflict here since "chef-dk ruby will take over my distro" after running chef shell-init

i think the point of doing this, however, is that people who want to opt-out of that behavior should use chruby/rvm/rbenv in order to do normal ruby-switching using /opt/chef/bin as an external ruby directory -- and they should not be using chef shell-init at all.

[lamont-granquist]

Yeah it looks like the only thing we lose is that chef shell-init how also loads up completions for your shell so you'd lose those. The rest of chef shell-init i think is just overriding the PATH and GEM_* env vars similar to what rvm, et al. do.

[docwhat]

@lamont-granquist you were the one who agreed with me that the embeded/bin should not be in the PATH.

It's not because of the ruby take over. That's fine and I and at least two others use https://github.com/docwhat/rbenv-chefdk to switch.

The problem is OS specific tools like tput and xml-config which can really mess up compilation and other scripts.

[lamont-granquist]

and libtool like in chef/chef-dk#682.

yes, i'm just thinking through all the implications, because in chef/chef-dk#854 people are upset that we're putting git in their PATH.

now i think this is the correct course of action to fix that by putting git in embedded/bin and taking embedded/bin out of the PATH.

but in the process we must move ruby into /opt/chefdk/bin/ruby and chef shell-init will put that into the user's PATH, making it seem like this might be creating a cure that is worse than the disease. i'm just writing up my thoughts on why that isn't true.

i think the solution there is that chef shell-init needs to be the API where you are explicitly asking for chef-dk to take over your ruby.

by doing this, though, we enable proper rvm and chruby switching to /opt/chefdk/bin -- so those users should simply not use chef shell-init -- or we need an option to chef shell-init to only pick up the command completion and none of the env vars -- chef shell-init --im-using-my-own-ruby-switcher

[docwhat]

Optionionally create a new bin for system clobbering executables, like git and such.

[lamont-granquist]

we don't need a third thing, we just need to use the two that we have correctly. we also have to move the ruby bins to /opt/chefdk/bin to make them play well with rvm/chruby/rbenv/etc.

[mrjcleaver]

Related: https://discourse.chef.io/t/chefdk-inside-docker-for-mac-multiple-rubies-rbenv-is-painful/8839

[lamont-granquist]

So the requirements we have are:

1. Be more rvm/rbenv/chruby-friendly
2. Pollute the system environment less with tools like libtool, pkg-config, etc which break homebrew and other compilation systems (including rvm native gem installs like in chef/chef-dk#682)

In order to meet the first requirement we need a directory to feed to rvm/rbenv/chruby that has the appbundled binstubs in them along with the ruby binaries (and any other installed binstubs that have been installed manually after-the-fact). This is basically the requirement that we have a consistent "whole" ruby environment. We have a few choices:

1. /usr/bin -- this is obviously bad since we don't want to blow away the system ruby, but its just for completeness
2. /opt/chefdk/bin -- this is the solution that i favor
3. /opt/chefdk/embedded/bin -- this is problematic and leaves /opt/chefdk/bin somewhat pointless
4. some additional location -- i don't feel there's any need

The /usr/bin location is just offered as an observation that we already have "minimally invasive" way to install omnibus packages onto the system, which is just to have the appbundled stubs exposed through the symlinks in /usr/bin and in the $PATH that way. This is the way that chef-client is typically invoked. The current utility of /opt/chefdk/bin is limited because it duplicates a use-case that is already in the system. It is only mildly useful for dealing with systems that have both chef and chef-dk installed, but the other cases will also have this covered.

The /opt/chef/embedded/bin solution is problematic because we'd need to change all the existing binstubs to appbundled binstubs, we'd also need to move the utilities like git, tar, pkg-config, libtool and everything else out of that directory. After doing all that work, /opt/chefdk/bin becomes a pointless directory that nobody would want, and would always be added to the PATH whenever embedded/bin was, and would always either shadow or be shadowed by all the binstubs in embedded/bin. The level of user disruption is also not mitigated at all by this approach. We still have to take utilities like libtool, pkg-config, etc out of the $PATH to meet the second requirement, so anyone who has built dependencies on having those tools by default still need to change the environment to add whatever other path we move them to.

The solution of adding yet-another-directory is also problematic for all the reasons that the embedded/bin solution was problematic. You still have to take embedded/bin out of the $PATH in that situation in order to accomplish the second requirement, and users still need to add it back manually, the level of disruption is the same, and now there's even more directories to manage.

The /opt/chefdk/bin solution limits the amount of things that we have to manage, and have to teach to users, while having identical amount of potential disruption all the prior to examples. In this case /opt/chefdk/bin becomes a "full-featured" ruby directory that rvm/rbenv/chruby can consume which will have appbundled bins along with the ruby binaries and irb and other utilities and any other binstubs that have been installed.

There is one open question of what to do about the binstubs and the ruby binaries that exist in embedded/bin. It will be critical that in order to keep the "whole ruby environment" requirement met that gem install whatever into the ruby will result in binstubs in /opt/chefdk/bin being created. It is also critical that we ship appbundled bins in /opt/chefdk/bin. I'm unaware of a hard requirement for what happens with the binstubs for the appbundled binaries that we currently ship in /opt/chefdk/embedded/bin or the ruby executable that we ship in embedded/bin. IDK if those stay as they are or become symlinks or hardlinks, or go away. We would expect that embedded/bin would only be added to the $PATH such that /opt/chefdk/bin would already be present (and /opt/chefdk/bin should really always come before the embedded/bin even today) so its not clear that its important at all to maintain the binstubs and ruby executables in this directory. That's is probably the biggest open question (which most likely needs a POC in order for us to kick the tires on how it works, and have an engineer struggle with trying to build it one way or the other, rather than debate--the answer will probably become clear as we build it).

There's also been a concerned raised over how this would affect native gem installs if the embedded/bin binaries are not available when the user does a gem install. However, the most likely utility that would be needed would be pkg-config to setup the LFLAGS, CFLAGS and LIBPATH/rpath in order to link against the embedded/lib that ship with omnibus. Ruby already has the mkmf.rb configuration which bakes in pkg-config-like functionality so that native gem installs will pick up those flags by default. The edge cases of native gems which don't install when those utilities are not in the PATH should be some small subset of (badly written) native gems. The trade off here will always be between these installs being correct vs. not meeting the second requirement and thereby completely breaking homebrew and native compilation outside of the chef-dk environment (injecting pkg-config and libtool into the $PATH is what we very much MUST stop doing).

[lamont-granquist]

My naive bias would be to remove ruby and binstubs from embedded/bin since that produces the simplest system overall. However, at this point I suspect that embedded/bin/ruby has been baked into far too many things, and we can't remove that. Some system of duplication/symlinks is probably required. It may be best to move the binaries to live in the /opt/chefdk/bin directory and make that 'primary' for ruby and install symlinks in embedded/bin.

IDK if we can make both of those be "fully-functional" so that any user-driven "gem install" adds binstubs into both directories. I suspect ruby cannot be configured that way?

[lamont-granquist]

So, I could probably be talked into option number 3 actually being better via going down the route where we realize that we need to have embedded/bin/ruby and can never change that, while we can't easily move it into /opt/chefdk/bin either (IDK why this would be though) and that it makes more sense to add binstubs to embedded/bin and move all the extra binaries out of that directory. That's pretty unpleasant, though, since it leaves /opt/chefdk/bin as a vestigial "why the fuck is this thing here?" directory -- which I think in the name of limiting user confusion we should try to avoid.

We could leave the appbundled stubs in /opt/chefdk/bin while having normal binstubs in embedded/bin, but then rvm/rbenv/chruby users will get shitty-slow knife commands when they put embedded/bin in the $PATH first and I think that solution fails to meet the requirement that we be rvm/rbenv/chruby-friendly. The acceptance criteria should be that we have a directory that has appbundled chef/chef-client/knife/ohai/etc, has ruby/irb/etc and has user-installed binstubs and nothing else.

[docwhat]

You could have two paths, one for users that only use the ruby in chefdk. The other for rmv users.

[lamont-granquist]

the chef --shell-init users, though, still have all the same kinds of requirements, only dropping the requirement that it all be in one directory. right now they're happy with having binstubs in /opt/chefdk/bin and ruby in /opt/chefdk/embedded/bin. they have the same unhappy sadness with having all the embedded/bin binaries that trash homebrew and other native installs when embedded/bin is currently in the $PATH. so they're very much affected by the second requirement here, while the first requirement also satisfies what they need. they're also not really exposed to the details of what directories are in their PATH (the actual directories there are not a public API) but to the overall behavior -- appbundled binstubs + ruby.

the other consumer here that i can think of is the consumer who actually has become dependent upon having the extra binaries in the $PATH, those consumers are going to have to be disrupted, and they do need another directory, which is at the core of the proposal -- the different options are basically around where that lands. I'm arguing that /opt/chefdk/bin becomes the directory that both rvm and typical chef --shell-init users use, which satisfies both their requirements. for the users which are disrupted by removing the embedded utils from their $PATH they would have to add back embedded/bin (and internally chef-client itself knows it needs to do this and does it via path_sanity).

i can also make a pretty good argument based on back-compat here. while we're taking away a directory from the $PATH and creating a bit of a breaking change due to removing some binaries from the $PATH (but that is a design goal of this proposal in order to meet the second requirement), we can fix this with only additive, minimal disruptions to the filesystem layout:

1. add ruby to /opt/chefdk/bin
2. add user-installed binstubs to /opt/chefdk/bin

we can leave everything else the same in /opt/chefdk/embedded/bin -- the binstubs, the ruby, and the extra programs. this ensures that any hard-coded paths to any of those will also resolve. this is likely a hard requirement for /opt/chefdk/embedded/bin/ruby, but also may be a requirement for other utilities.

by trying to move the utils out of embedded/bin then you are effectively deleting them and breaking those full paths to the binaries which may have become baked into software out there. i think that makes the third and fourth options worse compared to this, where embedded/bin needs to have these utils removed from them (i suppose the fourth version you could go nuts creating additional kinds of directories while keeping embedded/bin the same and winds up a situation where again you've got the 'why do we have four things when we only need two?' which i think makes it much less favorable).

so, i think i've just made a good argument that this approach is the likely the right one and is the most back-compat possible, while breaking back-compat only where we have to in order to implement the second requirement.

in summary:

• ruby binaries need to be added to /opt/chefdk/bin while also remaining in embedded/bin (which direction the symlinks run or how its implemented are implementation details)
• user installed gems need to be added to /opt/chefdk/bin while also remaining in embedded/bin (this is the tricky bit i think)
• appbundled binstubs should remain in /opt/chefdk/bin
• embedded/bin should be removed from the $PATH setup by chef --shell-init

that second point is the hard part -- don't know if ruby can be convinced to install binstubs in two different dirs. but if we don't we wind up with two different possibilities -- move user-installed binstubs only to /opt/chefdk/bin or else we wind up trying to use embedded/bin and both moving the appbundled binstubs there and moving the binaries out of embedded/bin and declaring bankruptcy on /opt/chefdk/bin. i think that if it comes to that i would pick that we still do this only user-installed binstubs move from embedded/bin to /opt/chefdk/bin -- we can still install ruby in both directories with symlinks and maintain /opt/chefdk/bin.

[lamont-granquist]

okay so about the last edge case i can think of is around the behavior of the appbundled binstubs when someone installs a gem that may overwrite them.

there's a problem, though, which is that the rvm users really need one directory which has appbundled binstubs + ruby or else we wind up back where we are now and don't solve anything for rvm users.

so to try to 'fix' the appbundled binstubs you keep them in their own directory like they are now:

1. /opt/chefdk/bin - appbundled binstubs
2. /opt/chefdk/ruby/bin - all the ruby binaries and user-installed binstubs
3. /opt/chefdk/embedded/bin - all the ruby binaries plus git and tar and the rest

that still fails to satisfy the first requirement because rvm users still need two directories in this model. there's a variant of this model where git gets moved elsewhere and embedded/bin becomes the ruby binaries + binstubs directory but that model also has the exact same problem, while being more invasive to existing full paths.

so an open question here is that as-proposed how do we prevent gem install chef from obliterating the /opt/chefdk/bin/chef-client binstub?

[danielsdeleo]

I think if you really want to have rvm or whatever and let it put its ruby bin dirs first in your path, then you have to accept that it might put knife ahead of the appbundled knife. These are just two completely incompatible requirements. IMO we should scope the solution to just fixing the case of the system-y executables breaking things.

[lamont-granquist]

you mean that rvm users don't get appbundled knife or that gem install chef manually may break appbundled knife? i'm more okay with letting the latter one break when people do things somewhat out of scope, less okay with the former one meaning that effectively rvm users don't get appbundled binaries at all.

[lamont-granquist]

i wonder if we could patch rubygems so that it exported a hook we could use for creating the binstubs, so that even 'gem install chef' would be appbundled?

[danielsdeleo]

effectively rvm users don't get appbundled binaries at all

If you gem install chef-client, then you'd get either the gem version or the appbundled version based on what's first in your path. If it's not the gem-installed version then you're potentially breaking someone's assumption about gem install chef -v VERSION; chef-client having a certain behavior. If the gem-installed version goes first in the path, it's probably gonna be slow. If you auto-appbundle it, then you're adding a lot of complexity we have to maintain and making it de facto supported to update components of the ChefDK and we'll get support questions about combinatorics of different versions of chef/berks/kitchen/etc.

[lamont-granquist]

so in this proposal 'gem install chef-client' results in 'upgraded, but slow', which doesn't break any major assumptions at least.

[docwhat]

Are we proposing changing it so that gem install <foobar> puts binstubs /opt/chefdk instead of ~/.chefdk/gem/ruby/?

Part of the disconnect with ChefDK is that it is recommended to add chef shell-init to your shell init files.

Maybe ChefDK should either work in one of two modes:

1. As a sub-shell: chefdk starts a new shell (of whatever the user is using) with the correct environment.
2. As a part of rvm, rbenv, etc.

In both cases, the gems can be installed into /opt/chefdk someplace. We can hack the gem to refuse to install certain black-listed gems to prevent corruption (e.g. anything shipped in ChefDK as pre-installed).

On uninstall of ChefDK (and upgrade) it should remove all the user installed gems. I've had to nuke ~/.chefdk/gem before because something changed in a incompatible way inside ChefDK's ruby. Most gems installed are most likely from a cookbook, and should be easy to re-install as part of the normal development cycle.

I don't think using ChefDK (or even Chef Client) ruby as a replacement/default ruby is something that should be supported because it is hard to support and prone to weird behavior on upgrade.

[danielsdeleo]

installing to /opt/chefdk isn't a good option because it requires root.

The issue where you have to nuke ~/.chefdk/gem isn't related to where it is, what happens is that some of the gems in there have dependencies "disappear" because the dependency was in the old chefdk that got removed by the upgrade. Definitely something we need to fix, but it's orthogonal to what we're talking about here.

[docwhat]

I assumed we'd fix the "requires root" problem some how. e.g. creating a mode 1777 directory where each user can put their own gems or something. Something like /opt/chefdk/usergems/<username>/... where /opt/chefdk/usergems would be mode 1777.

Optionally, create a group (chefdk) and make (using my example above) /opt/chefdk/usergems be mode 1775 and owned by root:chefdk.

The issue where you have to nuke ~/.chefdk/gem isn't related to where it is, what happens is that some of the gems in there have dependencies "disappear" because the dependency was in the old chefdk that got removed by the upgrade.

Right. But if we stored it in (from my example above) in /opt/chefdk/usergems we can nuke that directory on uninstall and upgrade via the package installer.

This is just an idea, BTW. I won't be offended if it is thrown out.

[lamont-granquist]

Are we proposing changing it so that gem install puts binstubs /opt/chefdk instead of ~/.chefdk/gem/ruby/?

I've been working on chef/chef too long, I totally forgot about that...