arkade Determine latest versions for tools not hosted on GitHub

Determine latest versions for tools not hosted on GitHub

Expected Behaviour

To lighten the load, kubectl and the various HashiCorp tools which are not hosted on GitHub, could determine their latest (stable, not pre-release) versions for download.

Current Behaviour

We hard code a few of these, but GitHub uses auto-determination through the releases HTTP endpoint.

Possible Solution

Do HashiCorp provide an API or HTTP endpoint for this? Or a static file somewhere?

For kubectl, Google provide one: i.e. curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

Steps to Reproduce (for bugs)

Remove the version for one of these tools
Run arkade get
See it pull in the latest stable version, and not a pre-release or alpha

Oct 23 '21 08:10 alexellis

Neither Kuma (#571, #572) provide GitHub releases. But it also has an endpoint for it: https://kuma.io/latest_version.

Should we introduce a new field in the Tool struct for example VersionURL, if provided and Version is not, then fetch the provided URL and use the response as the version string?

Oct 30 '21 10:10 Shikachuu

After some research, I was yet to find any HashiCorp related release version urls, every available script that trying to achive the same goal was parsing their GitHub releases. So basically something like this should work:

curl https://api.github.com/repos/hashicorp/terraform/releases/latest | jq .tag_name | sed 's/^.//;s/.$//'

Oct 30 '21 10:10 Shikachuu

That is looking very promising. For the version checking we are better to use the HTTP HTML endpoint than the JSON API request since it's not rate-limited.

Can you suggest how you would like to handle this at runtime, so we don't need to pin a version?

Nov 04 '21 19:11 alexellis

This might be a good idea: https://github.com/alexellis/arkade/issues/574#issuecomment-961342358_

Or this one: https://github.com/alexellis/arkade/pull/574#issuecomment-961340135

(Just to keep these ideas one place.)

Nov 04 '21 19:11 Shikachuu

What do you think about changing the Version field from a string to a function returns a string? Then, each tool can be implemented according to its requirements. https://github.com/alexellis/arkade/blob/19077193030259be439587b2686b0d4530ced248/pkg/get/get.go#L35-L37

Current code would be transformed to the following schema (so nothing would be affected)

			Version:     func() string {return "1.6.13"},

more complex scenarios could be like this

			Version: func() string {
				resp, err := http.Get("https://kuma.io/latest_version")
				if err != nil {
					return ""
				}
				body, err := ioutil.ReadAll(resp.Body)
				if err != nil {
					return ""
				}
				return string(body)
			},

or even with an error returned for better error handling

			Version:     func() (string, error) {return "1.6.13", nil},
...
			Version: func() (string, error) {
				resp, err := http.Get("https://kuma.io/latest_version")
				if err != nil {
					return "", errors.Wrapf(err, "Failed to get kuma website")
				}
				body, err := ioutil.ReadAll(resp.Body)
				if err != nil {
					return "", errors.Wrapf(err, "Failed to read response")
				}
				return string(body), nil
			},

Nov 06 '21 19:11 YuviGold

What do you think about changing the Version field from a string to a function returns a string? Then, each tool can be implemented according to its requirements.

https://github.com/alexellis/arkade/blob/19077193030259be439587b2686b0d4530ced248/pkg/get/get.go#L35-L37

Current code would be transformed to the following schema (so nothing would be affected)
			Version:     func() string {return "1.6.13"},
more complex scenarios could be like this
			Version: func() string {

				resp, err := http.Get("https://kuma.io/latest_version")

				if err != nil {

					return ""

				}

				body, err := ioutil.ReadAll(resp.Body)

				if err != nil {

					return ""

				}

				return string(body)

			},
or even with an error returned for better error handling
			Version:     func() (string, error) {return "1.6.13", nil},

...

			Version: func() (string, error) {

				resp, err := http.Get("https://kuma.io/latest_version")

				if err != nil {

					return "", errors.Wrapf(err, "Failed to get kuma website")

				}

				body, err := ioutil.ReadAll(resp.Body)

				if err != nil {

					return "", errors.Wrapf(err, "Failed to read response")

				}

				return string(body), nil

			},

Such a great idea! @alexellis what do you think?

Nov 06 '21 20:11 Shikachuu

Let me get back to you on this. Thanks for looking into it

Nov 19 '21 09:11 alexellis

Hi @YuviGold

Changing Version from a string to function would involve a massive refactor. I think we could probably add a separate field with a different name like: VersionSource

When empty, VersionSource would default to the current behaviour. If a string was given, i.e. kuma hashicorp then we'd determine what code to run (some custom lookup function) and return the version.

Wherever Version is used, would need to be updated to read from the specific version lookup function.

Nov 22 '21 14:11 alexellis

Hi @YuviGold

Changing Version from a string to function would involve a massive refactor. I think we could probably add a separate field with a different name like: VersionSource

When empty, VersionSource would default to the current behaviour. If a string was given, i.e. kuma hashicorp then we'd determine what code to run (some custom lookup function) and return the version.

Wherever Version is used, would need to be updated to read from the specific version lookup function.

This approach creates a bit more complexity(conditions) and misses the flexibility of the former. For example, if one source would return a JSON, or some other response, instead of a raw data. That being said, I agree that refactoring the function is a bit more tedious but not much. As long as there is no other package that uses this API as a library. Please correct me if I'm wrong.

Anyway, both implementations sound good to me :)

Nov 22 '21 19:11 YuviGold

Hi @YuviGold Changing Version from a string to function would involve a massive refactor. I think we could probably add a separate field with a different name like: VersionSource When empty, VersionSource would default to the current behaviour. If a string was given, i.e. kuma hashicorp then we'd determine what code to run (some custom lookup function) and return the version. Wherever Version is used, would need to be updated to read from the specific version lookup function.

This approach creates a bit more complexity(conditions) and misses the flexibility of the former. For example, if one source would return a JSON, or some other response, instead of a raw data. That being said, I agree that refactoring the function is a bit more tedious but not much. As long as there is no other package that uses this API as a library. Please correct me if I'm wrong.

Anyway, both implementations sound good to me :)

I think, the version that @alexellis recommends, can be introduced without any code change on the arkade/pkg/get/tools.go side, which is currently one of the most frequently updated files, not to mention it is not a breaking change, so it's just a minor bump.

That being said, your version still the elegant way to do it, yet that is a major version bump and introduces a lot of change in the above mentioned file and it's tests.

Nov 22 '21 19:11 Shikachuu

Hi,

Sorry, I didn't read this conversation before doing my PR (https://github.com/alexellis/arkade/pull/602).

I have done a little experiment to retrieve the last version using Github API. Any feedback are welcome :)

Jan 13 '22 19:01 Yannig

Sorry for the other issues. I will continue the discussion here.

In the case of tools like kubectl and helm, they don't have binaries directly on Github. It is therefore necessary to specify the version in order to be able to download the binary. On the other hand, in the case of these projects, the code is present in Github and it is possible to retrieve the latest known version using the findGitHubRelease function.

My proposal would therefore be to make the request even when the release does not contain the Github address (https://github.com/). Here the code to modify inside pkg/get/get.go:

	if len(version) == 0 &&
		(len(tool.URLTemplate) == 0 || strings.Contains(tool.URLTemplate, "https://github.com/")) {
		log.Printf("Looking up version for %s", tool.Name)
		v, err := findGitHubRelease(tool.Owner, tool.Repo)
...
		version = v
	}

We can change the if condition for the following one:

	if len(version) == 0 &&
		(len(tool.URLTemplate) == 0) {
		log.Printf("Looking up version for %s", tool.Name)
...
	}

Alternatively, one can also use a special version value to force the download of the latest version (like latest for exemple). Here the corresponding condition:

	if strings.Compare(version, "latest") == 0 || len(version) == 0 &&
		(len(tool.URLTemplate) == 0 || strings.Contains(tool.URLTemplate, "https://github.com/")) {

We can thus use a specific value in the tools.go file while leaving the possibility for users to download the latest version.

In fact, I think it's the only thing to change inside the code of arkade to get this feature.

What do you think ?

Jan 19 '22 18:01 Yannig

There are tools which do not use GitHub at all for their releases or binaries, but separate text files or HTTP endpoints.

There's been some discussion with @Shikachuu and my preference would be to implement an optional field like "source" = "github" (or "" blank as a default) / "hashicorp" / "kubernetes" etc for these non-standard tools.

At runtime, the value of this field would use a "VersionFinder" interface to determine the URL.

For the GitHubVersionFinder, the existing code would be used. For Hashicorp, their series of text files / HTTP endpoints, and likewise for kubectl:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

For Hashicorp, there is a download server that can be browsed, but there may also be a text file that can be read. I'm not sure:

https://releases.hashicorp.com

If you're certain that every downloadable tool we maintain in arkade has a release on GitHub then perhaps we can adapt the existing code.

Jan 20 '22 10:01 alexellis

It looks like there are releases on Github for most of the tools. So far I have only found hashicorp/vagrant, minio/mc and rancher/kim. For the rest, it seems to be there.

Jan 20 '22 18:01 Yannig

Just checked about the different URLs

Click to expand the code:

func Test_Tools(t *testing.T) {
	tools := MakeTools()

	nonGitHubURLs := make([]string, 0)

	for _, tool := range tools {
		url, err := tool.GetURL("", "", "")
		if err != nil {
			t.Fatal(err)
		}

		if !strings.Contains(url, "github") {
			nonGitHubURLs = append(nonGitHubURLs, url)
		}
	}

	fmt.Println("github URLs: ", (len(tools) - len(nonGitHubURLs)))
	fmt.Println("Non-github URLs: ", len(nonGitHubURLs), nonGitHubURLs)
}

github URLs:  67
Non-github URLs:  8 [https://get.helm.sh/helm---amd64.tar.gz https://storage.googleapis.com/kubernetes-release/release//bin//arm/kubectl https://releases.hashicorp.com/terraform//terraform___.zip https://releases.hashicorp.com/vagrant//vagrant___.zip https://releases.hashicorp.com/packer//packer___.zip https://releases.hashicorp.com/waypoint//waypoint___.zip https://dl.min.io/client/mc/release/-/mc https://dl.influxdata.com/influxdb/releases/influxdb2-client---.tar.gz]

I believe that a mix of VersionSource + Having a flexible Version function is the optimized solution. But we can start only with the first one to avoid refactoring. @alexellis let me know if you want me to implement any of the solutions.

Jan 27 '22 14:01 YuviGold

To put a hacky bandaid on the HashiCorp tools we can call the github API since they tag their releases there and after that just use the provided url template.

This should be a minor change in the runtime in this condition to be specific, yet as I mentioned above this is a hack. Yet to be honest, at this point I trust the documented and versioned GitHub API more then the "random" download server that HashiCorp provides without versioning.

wdyt @alexellis?

Jan 02 '24 21:01 Shikachuu