Pauld/1559705 improve cli to detect viable sdks based on debian flavor

[pauld-msft]

Hi all, sorry for the monster of a PR. Luckily, a lot of it is small changes. This is the first of 3 pull requests, which are documented at the links below.

Link to ticket: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1559705 Link to high level plan: https://microsoft-my.sharepoint.com/:o:/p/pauldorsch/EjlwXiCawbVJpdhAIrlGwEIBKF43t43jgXebfVuTh0fXJQ?e=vo1VZG

TLDR; PR 1 (This one): Updates what we upload to the sdk storage, only lets cli pull sdks for its debian flavor, and pushes some bullseye sdks. PR 2: Updates logic to use bullseye sdks, which won't exist in the storage account until we merge this PR. PR 3: Release some new bullseye-only sdks (like Node 18, Python 3.10, and Python 3.11)

---

Official changeleog -

SDK Storage Upload

• [x] Updated file structure to support specifying versions for each os type

  • Image of new file structure

• [x] Storage accounts will now contain default files like defaultVersion.<debian flavor>.txt

• [x] Add metadata to each item we upload

  • SDK_Version
  • Os_type
  • Version (only stretch images will be uploaded with this, it is what is currently used to generate the list of compatible versions)

• [x] TODO: determine what SDKs we want uploaded for each os type

Shared code generator

• [x] Support for new file structure
• [x] Fixed bugs with supported platforms documentation file generation, so we can now keep all changes that come from executing the generateConstants.sh script
• [x] Support for a legacy versions file, which allows us to keep the supportedVersion.md file up to date automatically

CLI

• [x] Support for platform installation only pulling sdks that are associated with its current debian flavor
• [x] Added regex to support determining compatible sdks (for non-stretch) for now, to be deprecated once the sdk metadata is uploaded to the sdk storage

Tests

• [x] Update to storage account tests to make sure that we have the expected versions and files
• [x] Update cli tests to ensure that debian flavor is being respected in what is being pulled

Small enhancements

• [x] Allow for buildPlatformBinaries.sh to move tar files into artifacts directory for node and java when executed on windows machine
• [x] Added php fpm config to constants file and made relevant changes so they aren't overwritten when generateConstants is run

[pauld-msft]

One of the big questions that I want to address here is the TODO from the PR description. Specifically, determining the versions to upload for each SDK type. I would lean towards keeping all of the current versions that we are supporting in the storage account, for backwards compatibility reasons.

Furthermore, there is definitely a set of bullseye sdks that we will need to have for building the images but I don't have the full list unfortunately. An example sdk that we will need is Node 12, for when we attempt to build the bullseye flavor azure function image https://github.com/microsoft/Oryx/blob/main/images/build/Dockerfiles/azureFunctions.JamStack.Dockerfile#L15

Looking for the team's input as to whether we should:

1. add the same images that currently exist for other non-stretch flavors, which is likely overkill but will ensure that we can build all of the images
2. spend the time to determine the minimum list of sdks needed to build bullseye based images, and only add those to the sdk storage account

Regardless of what we choose, in the future we can be intentional about what OS types we decide to release SDKs for.

[pauld-msft]

Regarding https://github.com/microsoft/Oryx/pull/1484#issuecomment-1190304289, @daniv-msft and I had an offline conversation about it and we are going to try to keep the sdks that we upload for buster to a minimum. For now, we are trying to just upload the ones that we know we will need for runtime, and I made a change to the jamstack build image so that it uses the supported node 14 instead of node 12, which all OS's have an associated sdk for.