gvt

gvt copied to clipboard

Per our discussion on twitter, I proposed creating a new command which would allow the user to vendor third-party dependencies in an existing project without needing to run "gvt fetch" for each.

Command Details

Name: imports Flags:

• no-recurse - don't recursively fetch dependencies
• precaire - allow insecure protocols

Usage: imports [-precaire] [-no-recurse] Description: Scan the project's source files for third-party imports and fetch/add them to the manifest file if they do not already exist.

Notes

Both flags have the same usage as in the existing fetchcommand. The current problem that exists is this tool only reads strictly from the manifest file (as does gb-vendor). I think that the go/ast package an be utilized on *.go files in the project directory (recursively) to get all imports. Logic for determining imports that need to be fetch'd can be referenced from the go get command here (see the download func).

That same logic could also be used for extending the normal fetch command to support repositories which do not have a manifest file.

:+1:

Two points: I don't think no-recurse makes much sense and I think imports should always work with or without the manifest, with or without stuff in vendor/ (it's ok to just nuke it), while all the other commands should be allowed to rely on the manifest.

That way imports would always be the way to rebuild the manifest, and the other commands can stay simple.

Got it. Rebuilding makes more sense since dependencies without the manifest file can just have their dependencies added directly to the project's manifest and makes that adding/dropping process a lot easier. I'll see about getting started on this soon (tonight/this weekend).

Quick status update on this. I pretty much have this command completed, but there are a couple of hiccups. Here is a diff of the changes so far (with some fmt.Print*s in there for debugging) .

Hiccups:

• Running the command on projects which reference a package that's part of their own project (e.g. this one references github.com/FiloSottile/gvt/gbvendor) causes the application to vendor itself. So in this case it vendors github.com/FiloSottile/gvt/gbvendor. I don't really know of a good way of preventing this except for splitting the package path and splitting it on "/", taking parts [3:], and checking if that subdirectory exists in the current directory
• Test files are checked as well. In the case of this repository we have _testdata which I've hardcoded to ignore, but it's an edge case still

~~I thought there was another but I lost my train of thought in the midst of writing the previous two 😛~~

• The third point was that when doing this you sometimes hit GitHub's rate limiting so I just sleep for 5 seconds if the package is located on GitHub to avoid hitting that.

Running this command on an existing project of mine produces the following manifest and file tree: https://gist.github.com/landaire/7dd69cc74419386da05b

I iterated a bit on this and this is the end result I'm implementing. Feedback very welcome!

migrate vendors the dependencies of an existing project.

This command has different behaviors based on how your project looks like:

    * if you were using Godep, the manifest file will be converted, the
      dependencies moved to the vendor/ folder, import path rewriting reverted
      and the Godep folder removed

    * if you were using Glide, the Lockfile will be converted, all other
      metadata removed (this drops semver and similar constrains)

    * if you were using any other tool, the vendor/ folder will be deleted and
      gvt migrate will proceed as if you were not vendoring dependencies

    * if you were not vendoring dependencies at all, gvt migrate will parse the
      imports tree and fetch the latest version of each external dependency

      NOTE: the latest version might not be what was in your GOPATH, and your
      build might break--this is the problem vendoring solves. If you need to
      vendor an old version, use gvt delete and then gvt fetch with
      [ -tag | -revision | -branch ]

Glide/Godep note: all migrated dependencies will be "headless"--only the
revision/commit is kept, and not what branch/tag/version it came from--so they
can't be updated without being deleted first.

It sounds good. Any progress or commit I can test?

I came to the issues to request this feature. Would be great to have it soon!

@stunti I haven't touched my branch in a while but this was what I worked on: https://github.com/FiloSottile/gvt/compare/master...landaire:feature/imports-cmd. See my comment above

@FiloSottile Hi, are there any updates from your side ? I was thinking of continuing what @landaire started and instead of fetching the repos directly just calling fetch(path, true) from fetch.go for each dependency. So the problems reduces to just getting the imports from the current root directory only ( which seems easy and already almost implemented by @landaire ) and fetch will do the heavy lifting and the recursion. Concerning the migration from other vendoring tools, I'd say we release the first beta version of gvt imports that only bootstraps from a package without any vendoring tools ( actually nukes them as you mentioned before ) and then we iterate on it in next releases adding migrations.

I wrote a quick and dirty command to use until the feature is released: https://github.com/jaschaephraim/gvtimports

@jaschaephraim thanks the tool was very useful. When are you planning on incorporating it directly into gvt?

You can also do this by leveraging the power of go list, along with some clever bash scripting. For example, the following bash script:

1. Finds all of the dependencies used by the package in the current working directory, including test dependencies
2. Compares that to the list of packages in the std library, filtering them out (since I assume you don't want to vendor the std lib)
3. Filters out any packages that are sub-packages of the current package, since you don't need to vendor what's already included in the repo
4. Calls gvt fetch on the remaining packages in the list

go list -f '{{join .Deps "\n"}}' > deps.txt
go list -f '{{join .TestImports "\n"}}' >> deps.txt
sort deps.txt -u -o deps.txt
go list std | sort | comm -23 ./deps.txt - | grep -v `go list` |  xargs -I{} gvt fetch {}
rm ./deps.txt

It has to create a temporary file (deps.txt) as part of the process, but it removes that afterwards.