imgpkg

imgpkg copied to clipboard

Describe the problem/challenge you have Library request: Today the only way to leverage imgpkg in workflow automation in client products is to consume the CLI and rely on its stdout and stderr mechanisms to track the output and errors. These mechanisms are helpful from a human readable perspective but not so much from an automation perspective since it

• requires parsing of the free form text which are not bound by a contract to track responses and errors
• forces clients to rely on system calls from inside their automation framework
• it reduces the scope for using the parallelization constructs the framework may provide

Describe the solution you'd like Provide client libraries in one of more languages, preferably Go and Java to help build automation around it, preferably as an API SDK library to enforce the contract.

Anything else you would like to add: If there is a need to prioritize the languages for the libraries, the preference is Go followed by Java.

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

cc: @ncdc

Thanks for the issue.

My initial reaction is that this would involve a lot of effort. In terms of developing the SDK in go and java. But also it introduces another entrypoint to how folk can use imgpkg, and hence, increases the different ways things can go awry. potentially increasing the support/maintenance load.

So, i'd like to examine the reasons you listed as to why the CLI isn't a good fit for the workflow automation, and see if we can identify changes to the CLI to address them, before undertaking the bigger plunge of a couple new SDKs.

requires parsing of the free form text which are not bound by a contract to track responses and errors

what kind of things are you parsing from the free form text? I'm assuming digests?

imgpkg has a --json flag. So there is the possibility of parsing a structured format instead. We could think about adding fields to the json response that you are currently relying on. i.e. in addition to .Lines we could have a .bundleDigest field too.

forces clients to rely on system calls from inside their automation framework

Hmm, even if we provide a golang SDK for imgpkg, system calls are going to continue to be made every time we open a file or write data to a network device etc. I am not sure how a SDK changes this dependency.

it reduces the scope for using the parallelization constructs the framework may provide

Currently the copy command provides a concurrency flag. The pull command currently does not. However we could introduce a concurrency flag to that command too. This concurrency flag is used to control how many images are downloaded and uploaded from/to a registry. This is also where most of the time is spent. Were you thinking of maybe running other tasks in parallel? If so which ones?

But also it introduces another entrypoint to how folk can use imgpkg, and hence, increases the different ways things can go awry. potentially increasing the support/maintenance load.

It's fairly common for tools written in Go (and probably other languages as well) to have the core functionality implemented in packages that are reusable by both the project itself and external consumers. You can also architect your command structure (e.g. with Cobra or a similar framework) so that the command structs are reusable as well. This minimizes possible variations across different entrypoints and provides the most flexibility. It's likely also easier to test tools that are architected this way.

Hmm, even if we provide a golang SDK for imgpkg, system calls are going to continue to be made every time we open a file or write data to a network device etc. I am not sure how a SDK changes this dependency.

I'm pretty sure @ashvinmoro meant an exec call to an external executable here. That is avoided if imgpkg's functionality can be invoked as a library.

If a tool such as imgpkg isn't available as a library, then the tool has to be available on the filesystem so a consumer can execute it. This has to be maintained using a different mechanism than importing Go code, which isn't always desirable.

Thanks @ncdc

I'm pretty sure @ashvinmoro meant an exec call to an external executable here

Can you please help me better understand the pain / issue around this exec call? Currently I'm making an assumption it is a performance concern. Which for a tool where it is common for operations to take several minutes, I don't see the benefit on shaving off sub-seconds.

then the tool has to be available on the filesystem so a consumer can execute it... different mechanism than importing Go code

There ultimately is going to be an executable that calls out to imgpkg, whether via a library or external binary. So there will be a mechanism on getting a binary deployed. However, it is sounding like, depending on your CI/CD setup, managing external binaries can come with some difficulty/complexity.

It would be useful to hear your thoughts on providing a json output instead of parsing free form text. This would be a much smaller change that ideally would help y'all consuming imgpkg in your workflow.

Here's hopefully some good news though!

imgpkg is currently written in go. It does use cobra and provide a cmd package that can be consumed and treated as a programatic equivalent to the commands exposed by the cli.

i.e. check out https://github.com/vmware-tanzu/carvel-imgpkg/blob/develop/pkg/imgpkg/cmd/copy.go to programmatically perform an imgpkg copy

While this is technically possible, there is currently no team process in place for 'supporting' this use-case. For e.g. We may (and have) performed major breaking change refactors around the API, and do not follow semver around these breaking API changes.

There is also no process around how to triage issues when a user is consuming imgpkg as a library vs a CLI. (Do we prioritize both issues equally, or do we use our engineering effort to work on the CLI issues instead)

As for the java sdk, that is a much larger undertaking. Given that fact, I would like to see some evidence, supporting a strong 'want' from the community.

@aaronshurley

I think this issue can be split into 2 issues.

1. Discussing a process to support imgpkg as a go library
2. Discovering whether there is enough signal to develop a java imgpkg SDK

I'm going to carvel-accept this issue, meaning we plan on working on it. When we do pick up this issue, we can then further define / create those 2 separate issues.

Can you please help me better understand the pain / issue around this exec call? Currently I'm making an assumption it is a performance concern.

It's more about managing the file dependency and less about performance. If you're writing an app that consumes something like imgpkg, and you want to containerize your app, if imgpkg isn't available as a Go library, then you have to find a way to get the imgpkg executable file in your container. You don't have to worry about this if you're using imgpkg as a library.

It does use cobra and provide a cmd package that can be consumed and treated as a programatic equivalent to the commands exposed by the cli.

i.e. check out https://github.com/vmware-tanzu/carvel-imgpkg/blob/develop/pkg/imgpkg/cmd/copy.go to programmatically perform an imgpkg copy

@ashvinmoro please take a look at this and see if it would work for you.

While this is technically possible, there is currently no team process in place for 'supporting' this use-case. For e.g. We may (and have) performed major breaking change refactors around the API, and do not follow semver around these breaking API changes.

I think this could be acceptable, although from a semver perspective it would be ideal if you could avoid making breaking changes in minor or patch versions. At the very least, I would recommend that you avoid removing functionality, but consumers can probably adapt to changing Cobra commands.

There is also no process around how to triage issues when a user is consuming imgpkg as a library vs a CLI. (Do we prioritize both issues equally, or do we use our engineering effort to work on the CLI issues instead)

I would guess that in the majority of cases, the same issue would be present in both the library and CLI cases, and that you would not necessarily need to distinguish between library and CLI when prioritizing.

Can you please help me better understand the pain / issue around this exec call? Currently I'm making an assumption it is a performance concern.

It's more about managing the file dependency and less about performance. If you're writing an app that consumes something like imgpkg, and you want to containerize your app, if imgpkg isn't available as a Go library, then you have to find a way to get the imgpkg executable file in your container. You don't have to worry about this if you're using imgpkg as a library.

This is our current scenario. We have to find a way to pull and containerezie the executable as part of our build process and also test for its presence. The other aspect is that the app will have to rely on running the command and on its return code from the underlying os which is a flaky coupling.

It does use cobra and provide a cmd package that can be consumed and treated as a programatic equivalent to the commands exposed by the cli. i.e. check out https://github.com/vmware-tanzu/carvel-imgpkg/blob/develop/pkg/imgpkg/cmd/copy.go to programmatically perform an imgpkg copy

@ashvinmoro please take a look at this and see if it would work for you.

I will take a look at this.

It would be useful to hear your thoughts on providing a json output instead of parsing free form text. This would be a much smaller change that ideally would help y'all consuming imgpkg in your workflow.

In the short term this is definitely desirable till we can get a "library".

An effort was started by creating the package https://github.com/carvel-dev/imgpkg/tree/develop/pkg/imgpkg/v1 which contains functions that allow other tools to use imgpkg as a library to do some operations. At this point not all the features are present.

Going to close this issue for now. If there are operations that you would like to have enabled via this API please reach out and we can add them.