Glide update does not update packages to the latest version unless git cache is cleared

[a13xb]

trafficstars

Your usage example states:

$ glide up                                # Update to newest versions of the package

However, it doesn't seem to do that at all. If I start with a clean repository with glide.yaml, where none of the packages are constrained, and a matching glide.lock, none of the dependencies are updated, even though upstream has new commits. Repository remains clean.

When (pre-0.12) Glide was still not stripping VCS files, I used to be able to go into venvor tree, checkout a specific version of a single dependency and run glide update, it would update the lock file to pin the dependency at the version I checked out. So I've been left with the impression that all glide update does is transfer the state of your vendor tree into glide.lock, that's it.

Obviously now vendor tree doesn't retain VCS information, I'm not sure what exactly glide update is supposed to do, and how to actually properly update dependencies to the latest version of the upstream.

[mattfarina]

I'm a little confused about something and I think I need some more information to understand what's going on.

The glide.yaml file holds the intent while the glide.lock file holds the specific revisions being pinned to. So, a version for a dependency in the glide.yaml file that's empty, tracks a branch, or has a version range causes Glide to go to the dependency source, pull down any updates, and store them locally. It does that automatically.

There was never the intent for someone to manually checkout a version inside the repo within the vendor/ folder. Glide tells the VCS (such as Git) to go out to the source and fetch updates.

This happens via the github.com/Masterminds/vcs package. For example, with Git it will drop down to git and use commands like git fetch and git pull expecting them to go out.

Is that not happening for you?

[a13xb]

Is that not happening for you?

When I run glide update, yes, it pulls the latest from the upstream VCS into cache, but the versions in the vendor/ workspace are not updated.

Let me clarify again: my glide.yaml is a plain list of dependencies, none of them specify a branch, a tag, a commit, or range of versions, none whatsoever. My expectation was that this will result in glide update always updating the glide.lock to the HEAD of master.

This doesn't seem to be happening. After the update of local cache, my glide.lock does not change, even though there are new commits in some upstream repos.

[INFO]  Replacing existing vendor dependencies
[INFO]  Versions did not change. Skipping glide.lock update.

There was never the intent for someone to manually checkout a version inside the repo within the vendor/ folder. Glide tells the VCS (such as Git) to go out to the source and fetch updates.

Yes, that was my misguided attempt to get a pinned version of a package updated in glide.lock because glide update would not do it otherwise.

[mattfarina]

@a13xb it sounds like there is a problem and one I'm not experiencing.

Can you provide an example project or glide.yaml file? And, can you share your OS/version and Go environment (go env)?

[a13xb]

OSX: 10.11.6 Glide: 0.12.0 (via Homebrew) Go: 1.7 Go environment:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/alex/workspace/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.7/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.7/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/73/jds74fnd03q8qkfyx_8zx8400000gn/T/go-build423096790=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

Sample project:

glide.yaml

package: github.com/a13xb/glidetest
import:
- package: github.com/gorilla/mux

glide.lock

hash: be6ffa009beb31b542f3a93c368fa8213aaf2fed7fc7e10914021349fe65ce9f
updated: 2016-08-31T23:38:13.074624366+10:00
imports:
- name: github.com/gorilla/mux
  version: 9fa818a44c2bf1396a17f9d5a3c0f6dd39d2ff8e
testImports: []

Commit 9fa818a in the lock file is from June, actual HEAD of master is 0b13a92.

Running glide update:

[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for github.com/gorilla/mux.
[INFO]  Resolving imports
[INFO]  Downloading dependencies. Please wait...
[INFO]  Setting references for remaining imports
[INFO]  Exporting resolved dependencies...
[INFO]  --> Exporting github.com/gorilla/mux
[INFO]  Replacing existing vendor dependencies
[INFO]  Versions did not change. Skipping glide.lock update.
[INFO]  Project relies on 1 dependencies.

[mattfarina]

Quick question, do you have github.com/gorilla/mux imported in your code? Glide code scans and follows imports found in the code to find the packages to use. If you have this in your glide.yaml file but didn't use it in your code it's dependencies won't be followed.

I setup an example with a file:

package main

import (
    "fmt"

    _ "github.com/gorilla/mux"
)

func main() {
    fmt.Println("test")
}

It fetched the dependencies.

Does this fix your problem?

[a13xb]

I've added this sample Go file and it adds the transitive dependency, so it completes the example, but the problem is still there.

I think it has something to do with cache. How to replicate:

1. Starting with the empty cache, do a glide update, the dependencies are properly updated, git repos in cache end up checked out at master.
2. Discard changes to glide.lock file.
3. Do a glide install, old versions from lock file get installed, cache repos are checked out at the versions in the lock file (detached HEAD).
4. Do a glide update, update won't happen and the cache stays pointing at the commit in the lock file.
5. Remove the cache, glide update will update correctly.

[karamani]

+1 I had the same problem.

[oschwald]

I had a similar issue. Removing the cache fixed it for me.

[vuleetu]

Same here. Seems like Glide does not switch to the latest version before export to vendor.

[KatelynHaworth]

:+1: This also seems to affect me.

Go Version: go version go1.6.2 linux/amd64 Glide Version: glide version v0.12.3 Go Environment:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/liam/.golib"
GORACE=""
GOROOT="/usr/lib/go-1.6"
GOTOOLDIR="/usr/lib/go-1.6/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

[jmazzitelli]

I confirm this is happening to me too. I was wondering if I was doing something wrong, until I read this issue :) I even removed the entry of my dep from the glide.lock file (my dep is on github), ran update again, saw that the .lock file put the dep back, but vendor/ is still the old version and the entry in the .lock file is still the old hash. I want to pull the lastest from master branch, but it isn't setting my dep to the latest hash - the lock file is showing the git hash from the first time I pulled the dep via glide. Workaround: "rm -rf ~/.glide/cache/src/my-dep-directory" then run glide update.

[adangert]

same problem I was having "rm -rf ~/.glide/cache/" worked for me, glide.lock was not updating to the latest commit hash regardless of doing a glide update

[sdboyer]

i'm pretty sure the answer is "no," but just to be sure: have any of you who are experiencing this bug done any direct manipulation of the ~/.glide/cache dir?

also, if anyone is able to create some exact steps to reproduce, it would be tremendously helpful in solving this problem.

[jmazzitelli]

@sdboyer the answer for me is "no" - in fact, I didn't even know this cache existed let alone where it was until I researched this issue and found someone said "clear you cache" and I had to go hunt for what this cache even was :)

[jmazzitelli]

@sdboyer

if anyone is able to create some exact steps to reproduce, it would be tremendously helpful in solving this problem.

I just ran through a set of replication steps for you to try.

FYI, my versions are as follows: $ glide -v ; git version ; go version glide version v0.12.3 git version 2.5.5 go version go1.7.1 linux/amd64

1: Clone this repo and make it your GOPATH: https://github.com/hawkular/hawkular-openshift-agent

2: Make sure glide is in your path obviously (to do this, I run "make install_glide" from the root GOPATH directory - there is a Makefile in there - and glide goes in my GOPATH/bin that I then run)

3: cd src/github.com/hawkular/hawkular-openshift-agent/ (you will find the glide yaml and lock files here)

4: Run "glide update" - notice this:

--> Fetching github.com/hawkular/hawkular-client-go.

5: See that you have the latest code for that dependency (hawkular-client-go) in vendor directory by grepping for "Username" - this was a new field added in the latest PR merge and is now in the HEAD of master branch:

$ grep Username vendor/github.com/hawkular/hawkular-client-go/metrics/types.go 
    Username    string

6: edit the glide.yaml and add a "ref" so that dependency gets locked on a new hash (this hash is the one before the latest PR merge and did NOT have that "Username string" field) - so add the two lines "vcs" and "ref" below under the hawkular-client-go package dependency:

- package: github.com/hawkular/hawkular-client-go
  vcs: git
  ref: d3a0e3ccb068e354fc61c5811a24414792e6159b

7: Run "glide update" - notice this:

--> Setting version for github.com/hawkular/hawkular-client-go to d3a0e3ccb068e354fc61c5811a24414792e6159b.

8: Run the grep again, and notice the Username is missing (this is correct, it means glide updated the dependency source under vendor properly)

$ grep Username vendor/github.com/hawkular/hawkular-client-go/metrics/types.go 
(...no output here - which is what you want...)

9: Now go back to glide.yaml, remove that "vcs" and "ref" that you added in step 6 thus bringing back your dependency to the master head.

10: Run "glide update" - notice this:

--> Fetching github.com/hawkular/hawkular-client-go.

11: Now, you are back to where we were (or we should be) - the latest code of master HEAD that has the new Username field in the dependency code. But run that grep command and see that it is missing under vendor directory:

$ grep Username vendor/github.com/hawkular/hawkular-client-go/metrics/types.go 
(...no output here - which is NOT what you want... we should be back on master and this Username field should exist)

12: To confirm the cache is now bad, go to it and look at what is checked out of git:

$ cd ~/.glide/cache/src/https-github.com-hawkular-hawkular-client-go
$ git rev-parse HEAD
d3a0e3ccb068e354fc61c5811a24414792e6159b

That is NOT the git hash of the latest master. That's the git hash we put in in step 6. But we want master, which currently is:

$ git rev-parse origin/master
4f5da7e951592445b686f95a0acdb0f860095d70

So when I removed the ref: from the yaml file and I "glide update", it never pulled the HEAD from master - the cache stayed on the old hash that I pulled before and thus my vendor/ never updated.

Hope this helps.

[sdboyer]

@jmazzitelli 👼 👼 ✨ 💖 🎉

i won't have time to actually run these through until later today or tomorrow, but these kinds of reproducibility procedures are like manna from heaven.

[jmazzitelli]

@sdboyer - any word on this? Did my replication procedures yield any clues as to the problem and potential fix?

[sdboyer]

sorry, i've had a ridiculous couple weeks, and just totally forgot about this 😞 i'll look today

[jmazzitelli]

@sdboyer those replicate steps might be slightly different - I just committed a change to that repo that changes the project structure to be more like a "normal" golang project. Just follow the README.adoc to install it (it should now be more as you would expect anyway). Its not really different, I just move the src/ files up to the top directory rather than committing the files under the src/github.com.... directory structure.

[sdboyer]

4: Run "glide update"

This is hitting another, i guess different issue:

[ERROR] Error scanning k8s.io/client-go/1.4/kubernetes: open /Users/sdboyer/.glide/cache/src/https-k8s.io-client-go/1.4/kubernetes: no such file or directory

it went away when I glide install'd, after which i could glide update successfully. I'll have to investigate this later, too...

12: To confirm the cache is now bad, go to it and look at what is checked out of git:

Yep, I figured the issue would be something like this. The git commands glide is issuing to get information from the repo aren't working correctly because they're using porcelain commands that have subtle dependencies on local repo state. Specifically, git pull doesn't do anything unless you have the right local branch checked out, and it's set up to track the right remote branch.

I need to track down where the actual calls are in the code to fix this up, but it shouldn't be too terrible to deal with.

Also, just a note for the future - the cache having a particular version checked out in the working directory isn't necessarily an indication that it's "bad". The technique we use to quickly get files onto disk under vendor/ is not copying what's checked out, but rather doing some under-the-hood git goodness to perform that process quickly. Right now, it's probably an accurate indicator, but in the future the working directory may be something totally different, but you'll still have the right thing in your vendor/.

[jmazzitelli]

Also, just a note for the future - the cache having a particular version checked out in the working directory isn't necessarily an indication that it's "bad"

This is why I have step 11 in my replication procedure. I actually do look in /vendor to see that the code is "bad" there.

[stevenroose]

I have the same issue. I track a package as a fork I made and when I push updates to the fork, glide update does not make the lock file update to the latest version of the fork. Also worked after cleaning the cache.

[sdboyer]

So, the fix over in #703 should address this. The only note to make is that you have to specify the branch you want to follow in glide.yaml - it won't move properly if you don't. (Yeah, I'm not happy about that either).

I've verified this against @jmazzitelli's replication procedure, but if someone else could try with that PR, that'd be awesome.

[jmazzitelli]

@sdboyer I just tried it using your PR branch and it still failed. But maybe I'm not doing it right. What do you mean "you have to specify the branch you want to follow in glide.yaml". I was using:

vcs: git
ref: d3a0e3ccb068e354fc61c5811a24414792e6159b

So I'm giving a hash reference, not an actual branch name. Is that not correct anymore?

And what about tags? Can you use tag names rather than branch names?

[franciscocpg]

@jmazzitelli I think this would be

vcs: git
version: d3a0e3ccb068e354fc61c5811a24414792e6159b

https://github.com/Masterminds/glide/blob/master/docs/glide.yaml.md

[jmazzitelli]

OK, I got it to work. The missing link is that YOU MUST specify "ref: master" when switching back.

If you just delete the ref: line, it will not work.

I tried with both version: and ref: and it all works but, again, it only worked IF YOU SPECIFY the master branch when you go back.

So these worked:

This was the commit that has the missing "Username" (see my replication procedures for what I mean about the "Username" field):

version: d3a0e3ccb068e354fc61c5811a24414792e6159b

or

ref: d3a0e3ccb068e354fc61c5811a24414792e6159b

Master has "Username":

version: master

or

ref: master

[sdboyer]

I tried with both version: and ref: and it all works but, again, it only worked IF YOU SPECIFY the master branch when you go back.

Righto, this is what I meant by:

the only note to make is that you have to specify the branch you want to follow in glide.yaml - it won't move properly if you don't.

ref is deprecated, please don't use that at all. (It will stop working entirely under gps, anyway)

[cep21]

Just to clarify, if I want to get the latest code when I run glide up I need to change my yaml to have version: master on every package like:

import:
- package: a
  version: master
- pacakge: b
  version: master
- package: c
  version master
- package: d
  version master

?

This seems like it should be the default. Maybe it's worth documenting that it's not.

[jmazzitelli]

@cep21 it is the default, the problem comes when you switch to another branch, and then want to go back to master (as you do when you are working in development and you want to pick up a bug fix in some package that hasn't been merged to master yet, then when it is fixed in master, you want to go back to master).. See my replication procedures above for what steps cause the problem.

[sdboyer]

The more I think about this, the more I think I need to go back in and see if I can make this work without having to specify the branch name. It's absolutely not acceptable that there would be two different outcomes to glide update depending solely on the state of the cache dir; in effect, that's changing the semantics of glide.yaml.