ASDF Plugin Standard Library

[ghostsquad]

Is your feature request related to a problem? Please describe

Looking at or developing plugins and seeing a lot of code repetition. It seems everyone has to start from 0 when creating a plugin. There are some really thoughtful plugins out there, like

https://github.com/kennyp/asdf-golang

and

https://github.com/asdf-community/asdf-hashicorp

such as in the ability to override the architecture for the install using an environment variable (e.g. ASDF_GOLANG_OVERWRITE_ARCH).

Describe the proposed solution

It would be really nice if there was a standard library or something like that so that plugins could be easily updated to include new features and that sort of thing.

Describe similar asdf features and why they are not sufficient

copy/paste... because... ya...

Describe other workarounds you've considered

I'd be willing to start such a project if this is not already being done somewhere. Please let me know. (Also, I would have posted this as a discussion if that was enabled).

Thank you for your time!

[hyperupcall]

This would be fantastic - I think one of the disadvantages of asdf is the lack of cohesion among plugins. There are some plugins that are exceptional, but others are lackluster, and it would be great to have some sort of facility to make it easier to make an exceptional asdf plugin.

In the interest of saving developer time, I might offer some functionality that could potentially be good as "standard library functions". I'm writing a similar tool to asdf, and I since I stumbled upon this issue, I thought it might be good to share some tools that made it easier for me:

• unpack() function for unpacking the more common tarballs (tar.*, zip)
• git_tag_to_versions_array() - A function for getting all the tagged commits of a GitHub repository, and putting that in a Bash array (because usually each tag corresponds to a particular version).
• fetch_github_release() - A function for getting ALL JSON release information about a GitHub repository. (Can be piped to jq or Bash for line-by-line processing)
• If you want to manage stuff like shopt and trap better between plugins and asdf code, I have a library for that and related things like stacktraces, but I'm not sure if that's as applicable for this codebase, as I see ample use of subshells

[ghostsquad]

@hyperupcall how would you feel about a go binary/library based system? Statically compiled, and no need to worry about if tools like jq and curl are installed or of a compatible version.

I think it's possible to go one step further even, and autogenerate code, released as a single binary. Think how hashicorp does their stuff, one plugin to manage a whole suite of tools they release.

Code generation could be made based purely on metadata, like a yaml file.

New tools or updates to tools simple get added to this file, which could be baked into the Go binary, eliminating any potential for binary/schema incompatibility, as well as ensuring an immutable supply chain (similar to a lock file).

[hyperupcall]

@ghostsquad Interesting idea - I know that Cash app has a tool called Hermit that seems to do what you describe. Their version management tool is written in Go, and they configure each code with a HCL file.

I'm not convinced that "baking everything into a single Go binary" is the way to go, though. You would have to integrate something like Cosmopolitan for the cross-platform binary, and it sounds unecessary complex, if I understand correctly

[hyperupcall]

I like the code generation / YAML and lockfile idea though - do you have any idea or example of what that might look like?

[ghostsquad]

It's not as complex as you think to cross compile go. GoReleaser is a tool that even helps make cross compilation declarative and super easy.

It may even be worth while at this point to fork ASDF core itself and rewrite it in Go natively. It would make it super fast, easy to test, and Go unlocks a lot of potential for advanced functionality.

It may actually be a method to eliminate the need for the plugin sprawl all together.

I am imagining folks going forward, could migrate to asdf-go and to ease the migration, for any given tool, check if that tool is in the new schema, and if it's not, fallback to current behavior (installing a distributed plugin).

[hyperupcall]

Hmm, it sounds like we were talking about different things with the compiling - I thought you were talking about compiling a Go codebase for all (relevant) platforms, to a single binary. As opposed to goreleaser, which easily allows for cross-compiling, as you mention.

And yeah, something Go-based would be something entirely different - but this issue is for a shell-based standard library?

[ghostsquad]

I'm not sure it needs to be shell based. I mean it could be, but the more i think about the problem, the more it seems that a shell based solution is more work than just writing the program in go from the start.

Hopefully you see what I'm saying. Like a shell based library will definitely reduce the code written for the majority of plugins, but my point is, what if you could just eliminate the need for those plugins by having a dynamic system for finding and downloading tools based on a metadata file. I certainly would not want to write such a system in shell. Having autogenerated binaries for major architectures, and making those available in standard ways, such as brew, chocolatey, apt, etc is likely not a huge undertaking.

[hyperupcall]

Right - yeah a shell based one is definitely more work compared to any modern language since most facilities are built in. Once I write my JSON parser for Bash, it would be good to have work on a declarative schema. 100% declarative would be ideal, especially since languages like Zig and Node have some sort of JSON file that can be transformed. But many languages don't, and some make it particularly hard like Dart. But even if it is difficult, I believe some declarative thing for everything would be a good end goal.

[amrox]

@hyperupcall I like the idea of a bash standard plugin library.

@ghostsquad I also like the idea of a super-fast, meta-data driven, go-based version manager, but I feel like that would be a fork or evolution of asdf as we know it.

[ghostsquad]

@hyperupcall I like the idea of a bash standard plugin library.

@ghostsquad I also like the idea of a super-fast, meta-data driven, go-based version manager, but I feel like that would be a fork or evolution of asdf as we know it.

Are you opposed to asdf continuing to evolve?

[amrox]

Are you opposed to asdf continuing to evolve?

Not at all.

Again, I like the idea of a next-generation or "Neoasdf". Honestly I'd like to work on something like that.

However imho, it will be easier to create a new project (or fork asdf), rather than try to steer this project if that direction.

[hyperupcall]

Again, I like the idea of a next-generation or "Neoasdf". Honestly I'd like to work on something like that.

@amrox If you want a "neoasdf", I would recommend looking into Woof. But, right now, it depends on a beta-quality Bash package manager, and several supporting Bash libraries that I have only tested on my machine. I plan on writing Rust version to be command-line compatable with the Bash version. I mention it to prevent duplicate work, and I'd also would like to hear your thoughts.

In any case, I think this would be good to implement in asdf, and may better resolve #1281 (compared to the existing PR)

[ghostsquad]

I can start a fork of the project and see what it would take to make this happen. Let me take a few days to get my thoughts on paper then invite contributions to the vnext of asdf plugins.

[daegalus]

Bit late to the conversation, but for the Go idea, you can use the model that Caddy uses. THe base caddy is pretty focused, but it has an expansive plugin system that compiles into your binary.

They then also released a webui to pre-build a combination of stuff you like, but also offer xcaddy which lets you create a config file of what plugins you want, and it rebuilds caddy whenever there is an update.

While we wouldn't necessarily need to do that, there are tools using similar models.

You could also use otto (JS) or some other embedded language, to write the scripts, that then go executes. Lots of options to flexibility.

[ghostsquad]

@Daegalus good ideas! My initial plan was to do something similar to terraform in order to avoid recompiling (download a binary, communicate over grpc). But even before doing that, I just wanted to look at what common list and install methods were being used and simply make something that allowed for just a declarative (yaml) instructions on where to go to download, what the plugin and binary should be named, etc so that no code (JS, Go, etc) is actually needed, just yaml.

[daegalus]

@Daegalus good ideas! My initial plan was to do something similar to terraform in order to avoid recompiling (download a binary, communicate over grpc). But even before doing that, I just wanted to look at what common list and install methods were being used and simply make something that allowed for just a declarative (yaml) instructions on where to go to download, what the plugin and binary should be named, etc so that no code (JS, Go, etc) is actually needed, just yaml.

That's totally a great approach too. It reduces the likelyhood of malicious scripts and other bad actor stuff while keeping it light and simple.

[hyperupcall]

@Stratus3D shall this be closed as well?

[ghostsquad]

@Stratus3D shall this be closed as well?

I think we should keep this open until work begins, as to help people find how/where to engage.

[hyperupcall]

In #1281, there was consensus for not implementing a "plugin standard library" - overall, it wasn't the preferred approach for this project.

This thread initially had some good suggestions with what an implementation might look like for asdf, but now the purpose of the thread has mutated to discuss a totally different project.

I personally do not see the value of keeping an issue open on the asdf repository to discuss something that is not related to asdf, especially since the original issue title has been resolved. By all means, feel free to create your own repository and brainstorm from there, the competition of ideas is great!, but I don't think this thread is the most appropriate place.

[ghostsquad]

The consensus of some seems to be that each plugin author should write the boilerplate, however, given these 2 issues alone, it seems that there's still desire to make asdf as easy as possible to use and contribute to.

If anything, both these threads are pointing to the desire to eliminate boilerplate if it's possible to do so. As an example, if 70% of plugins follow some standard way of listing and downloading binaries, why not encourage that golden path by taking care of all the boilerplate?

[jthegedus]

We have created the https://github.com/asdf-vm/asdf-plugin-template/ repository to handle the "boilerplate" for "most" plugins.

It has an interactive setup script to bootstrap and basic functionality required by asdf of a plugin. Yes, it is tuned for releases from GitHub and not "universal", but neither would an API in the core. It is the "golden path" starting point. Granted, updates to the implementation in the template aren't captured back into the repos that fork from it, so it might not be the right approach 2-3 years from now.

We (asdf core) are always thinking of improvements to the plugin API, but the discussions to date haven't lead to an outcome the asdf core are happy to pursue just yet. The discussions have been good and useful, but the cost/benefit isn't quite there for us to focus more on this part of the project, especially when the template we already have "works".

---

re GitHub Discussions

We tried GitHub Discussions for ~1 year and it just created noise. Few people ever went there to communicate or help one another. See - https://github.com/asdf-vm/asdf/issues/766#issuecomment-1039154909

[ghostsquad]

@jthegedus thank you for this thoughtful response. I have been thinking more about how these plugins, easy to install but essentially have no vetting process. Recently I've been thinking about the security concerns with the current approach.

[jthegedus]

@jthegedus thank you for this thoughtful response. I have been thinking more about how these plugins, easy to install but essentially have no vetting process. Recently I've been thinking about the security concerns with the current approach.

Yes, @Stratus3D and I are very aware of the security implications. We actually make asdf more difficult to use than alternatives because we require the user to perform manual setup as we don't want to encourage people executing code downloaded from the internet without first reading it and how it works.

The plugin repo model is going to continue until we get around to replacing it with a search model. I do read each repo for each plugin that we put in the plugin-repo, but someone can always add bad code later and I don't go back to re-read all commits on all ~600 repositories. We expect users to take responsibility for tooling they install. They should read it and know what it does.

There is a lot of work to do for this project. We're currently focusing on code quality and automation to improve the speed of our future efforts.

[dudicoco]

So what are the current plans for the future of ASDF?

I believe the suggested metadata file plugin system would solve the current usability issues of ASDF with plugins in external git repos:

1. They present a security concern and each one have to be vetted
2. They have to be installed manually rather than declaratively, making the .tools-versions file not as useful as it could be
3. Many of them are just duplicated code that downloads a binary from GitHub, it would be much easier to add a plugin by specifying which binaries to download in a metadata file