Problems with pivnet packaging and build-numbers

[aegershman]

Forgive me for these write-ups, but it's very useful to externalize them in pivotal-cf/om and make them visible for future references.

Starting in p-redis 2.1.3, the product version in opsmgr is no longer referenced as 2.1.3, it's using a -build.xyz number, referenced as 2.1.3-build.34.

Therefore, om cannot reference it using the product version it used previously in 2.1.2, which would be om stage-product --product-name p-redis --product-version 2.1.2

(the working command for 2.1.3 would be om stage-product --product-name p-redis --product-version 2.1.3-build.34)

Using the -build.xyz suffix isn't a problem, though. The problem is when the build suffix isn't available upfront as part of the product's distribution from pivnet, and only visible once it's been uploaded into opsmgr.

Notice in the case of pivotal-mysql, which uses build suffixes, the build suffix is available as part of the actual file name included in the pivnet bundle:

This allows automation to pull the build suffix pattern off the file so that tasks doing commands like om stage-product, configure-product, apply-changes, etc. can reference the "real" product version when it talks to opsmgr about pivotal-mysql 2.6.2-build.10. So for pivotal-mysql, even though the product version according to pivnet is 2.6.2, you can extract 2.6.2-build.10 from the file name, because 2.6.2-build.10 isn't included anywhere else like in the pivnet metadata, etc.

THEREFORE, in situations like p-redis, where the product version is listed as 2.1.3 or 2.2.0, even though the "real" product version is 2.1.3-build.34 or 2.2.0-build.90, since the product's file doesn't include the build suffix, there's no reasonable way to extract the actual product version:

The only way to get this data would be to either upload the product into an opsmgr, query for available-products, and then do regex parsing to get the build suffix...

# om --env ./env-pez19.yml available-products
+----------------------+----------------+
|         NAME         |    VERSION     |
+----------------------+----------------+
| p-redis              | 2.1.2          |
| cf                   | 2.4.11         |
| p-rabbitmq           | 1.16.3         |
| apmPostgres          | 1.6.1-build.12 |
| p-push-notifications | 1.10.3-build.1 |
| cf                   | 2.4.3          |
| p-redis              | 2.1.3-build.34 |
| pivotal-mysql        | 2.5.4-build.51 |
+----------------------+----------------+

OR doing something kind of funky by cracking open the p-redis-2.1.3.pivotal and doing a bunch of parsing logic to get to wherever in the *.pivotal bundle the "real" product version is stored, and then saving that off.

All in all, it's easier if the product version is stored in the product bundle's filename so it can be parsed out during the initial download from pivnet; or some variation where the product-version which other om tasks use can reference it.

Interested in thoughts people have on this, but feel free to close whenever.

Thanks!

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[nbconklin]

Thank you for sharing! I've also shared with the PivNet team and we will discuss further.

[calebwashburn]

@aegershman while there is definite room for improvement this is where the om tile-metadata command can help in the current spaces.

product_name="$(om tile-metadata \ --product-path product/*.pivotal \ --product-name)" product_version="$(om tile-metadata \ --product-path product/*.pivotal \ --product-version)"

This prevents you from having to first upload it or manually parse the tile metadata.

[rszumlakowski]

Is there a reason that the -build.nn text is being left out of the filename?

[kcboyle]

@rszumlakowski I believe it is up to the tile teams themselves to decide how their product appears on pivnet.

[iplay88keys]

@aegershman @calebwashburn @rszumlakowski Assuming we can't/won't have this fixed by tile teams or by pivnet, we have a proposal for a way to make the product name and version a bit easier to access:

Extend product-metadata(as the tile-metadata command name is deprecated) to allow getting the metadata directly from pivnet (implemented in a similar way to how config-template reaches out to read the metadata from pivnet directly). So, product-metadata would have the same output as we currently have, but you wouldn't need to download the product in order to get that information.

Thoughts?

[jaristiz]

@iplay88keys The solution you are suggesting involves making changes to PIVNET and the way the authors generate metadata, which is outside of the scope for this specific tool, if no further discussions we are going to close this.

[iplay88keys]

@jaristiz It's been a long time since I was on the Platform Automation team and we were managing om, but unless there have been changes in the past couple of years to how the product metadata files work, I believe this should be a viable solution for retrieving the Version of the Tile as defined by the Tile Developer in the Metadata file.

From what I recall, om config-template used a trick based on the fact that .pivotal files were really just .zip files, but renamed. It would download just the first portion of the tile in order to get the zip file's header file (which tells offsets for the contained files) and would use that to curl and download just the portion of the tile that contained the metadata.yml file.

Currently, om product-metadata allows you to retrieve the version from within the metadata file of a locally downloaded tile using the -product-version flag. Ex:

$ om product-metadata -p ~/Downloads/p-event-alerts-1.2.12.pivotal --product-version
1.2.12-build.2

Our suggestion was extend product-metadata to use the same mechanism that config-template uses in order to download the metadata.yml from the tile on Tanzu Network and then parse that to get the version directly. It would require adding the additional pivnet related command line options, but we already have the code for downloading the metadata.yml file from Tanzu Network, so I believe this would work.

The the resulting command could look like this, then:

$ om product-metadata --pivnet-api-token="<token>" --pivnet-product-slug=p-event-alerts --pivnet-product-version=1.2.12 --file-glob="*.pivotal" --product-version
1.2.12-build.2

[jaristiz]

Thanks for your reply.

I'll put this on our backlog so we can take a look and decide how much effort would be needed to make this a reality. For now, I am going to tag this one as enhancement and keep it alive.

[anEXPer]

I like this idea, but I'm cautious about using the presence/absence of Pivnet configuration as the implicit switch, as I believe it can also be loaded from env vars.

I'd prefer it with an explicit flag. This has a couple of small usability implications:

• Either the file path or the pivnet flag is required
• If the pivnet flag is present, filepath is not valid and should error
• If the Pivnet flag is present, the pivnet stuff is required from some source

I suspect this may complicate the validation quite a bit. Offering an alternative, I'd take an alternate version of the command, which might make for simpler validation logic. Either way's fine, just wanting to offer up easier paths if input validation gets dicey.

Nice to see you, @iplay88keys.

[iplay88keys]

@anEXPer Great points! I think that offers a better user experience. I'll work on the explicit flag option first and see how that turns out.

Nice to see you too!

[iplay88keys]

I submitted the PR. At this point, however, it is not accepting Environment variables directly for configuration options as it wasn't accepting them before this change, though it still works with a config file being interpolated by env vars:

$ export MY_api_token="<some-api-token>"

$ echo "pivnet-api-token: ((api_token))" > /tmp/config.yml

$ om product-metadata -c /tmp/config.yml --vars-env="MY" --pivnet-product-version="1.2.12" --pivnet-product-slug p-event-alerts --product-name --product-version
p-event-alerts
1.2.12-build.2

As such, it works like config-template in the fact that it keys off of the existence of pivnet flags and the product-path flag in order to determine where to grab the metadata file from.

[dtimm]

Nice addition! Merged.

Also, good to see you again @iplay88keys !

[iplay88keys]

Thanks! Good to see you as well!

[aegershman]

Hello from the future 👋 hope all is well. closing to prune open issues. take care all 🫡