microsoft
Use GitHub packages for internal dependency resolution
[!NOTE] Changes to the implementation made on December 18th 2023
Summary
Today, when having dependencies from one repository to another, you have to specify appDependencyProbingPaths with access to the other repository - or you need to create a direct download URL to a storage account containing your apps. Instead of this, we want to support GitHub packages for internal dependency resolution.
The following will utilize the functionality from https://github.com/microsoft/AL-Go/issues/261
If you setup a GitHubPackagesContext secret with access to packages for your org (or for another org), AL-Go for GitHub will automatically deliver every app to your private GitHub Packages NuGet registry named <publisher>.<name>.<appid>. AL-Go will also automatically locate apps in your private GitHub packages NuGet registry if the dependency isn't resolved by other mechanisms.
The GitHubPackagesContext secret content can be generated by the BcContainerHelper function called New-ALGoNugetContext like:
New-ALGoNugetContext -serverUrl "https://nuget.pkg.github.com/$myorg/index.json" -token $mytoken
Testing NuGetContext
NuGetContext successfully validated
{"token":"ghp_NDdI2Ewfsxxxxxxxxxxxxxxxxxx","serverUrl":"https://nuget.pkg.github.com/businesscentralapps/index.json"}
Dependencies to other apps will use the same mechanism as the NuGet packages for External dependencies if any of the apps contains a dependency on a 3rd party app.
Success Criteria
Just by creating an organizational secret called GitHubPackagesContext, all dependencies between your apps in all AL-Go repositories in that organization will auto-resolve.
How will it work?
The feature will use a fixed naming algorithm for auto-resolving dependencies - <publisher>.<name>.<appid>, but all package searches will ONLY use the appid. This ensures that you can rename the app publisher or app name and still resolve packages. GitHubPackagesContext should never point to NuGet.Org, as you potentially could download unsecure code published by somebody else.
The package name will be:
- <publisher>.<name>.<appid>
Other repositories in the same organization can automatically find packages from other repositories, simply by having the GitHubPackagesContext secret created under the organization.,
This feature utilizes functionality in BcContainerHelper 6.0.4 (in preview while writing this) and this is also used by the NuGet functionality for external dependencies.
See https://github.com/orgs/microsoft/projects/521/views/1?pane=issue&itemId=12395090 for details about the nuget package format etc.
Try it out
Run Update AL-Go System Files with freddydk/AL-Go@nuget to try out the latest version of this.
External vs internal
An automatic dependency resolution is not only desirable for internal dependencies but also for external dependencies.
Whether you are talking about maven, npm, pub, go or nuget.
These package mangers have no concept for internal or external.
They only know sources and the authentication used for them. (public vs authenticated)
We should have the same goal for dependencies in business central.
But there are some unique challenges in business central.
GitHubPackagesContext should never point to NuGet.Org, as you potentially could download unsecure code published by somebody else.
It's important to talk about the reason why this is dangerous and why this danger is unique to business central.
This is an organizational problem. The difference between dependencies in BC and for example Java, is only in how people decide which packages to depend on. You would only specify a dependency after it was deemed safe and reliable by a human. The foundation for that assessment is what’s in the public feed.
This is different in business central. You usually get your dependencies from somewhere, most likely from a website or via e-mail.
There are three requirements for a safe automatic dependency handling.
- Multiple feeds need to be searched when locating a dependency.
- Order is important. – You want to use your authorized feed before the public feed.
- Every dependency needs to be located before the public feed or the version from the public feed needs to be verified.
Requirements 1. and 2. are technical requirements which are relatively easy to implement. 3. presents a challenge because it would currently rely on human discipline. The user needs to make sure the package is resolvable from some authorized feed or verify the public package.
But there is a technical solution to this problem as well. Every partner already signs the apps that they create. Using this we can create a dictionary of trusted publisher name and public keys, from the code sign certificates.
When resolve a dependency you need to do the following steps.
- Requirements: Collect the trusted dictionary.
- Locate dependency in the feeds.
- Download the dependency.
- Get public keys for the vendor you specified in you app.json
- Verify the downloaded dependency to be signed with a private key that correspond to one of the public keys.
- Repeat steps 2-6 until you hit an unverified package or have no more dependency to resolve.
- Use apps as needed.
The advantages are the following:
- You have a fail safe when you add new dependency. If a dependency can’t be verified it won’t be used.
- When you verify a publisher all their packages will become trusted.
- By specifying appid and publisher in your dependencies (app.json) you decide which packages are THEIR packages. No trusted publisher can “hijack” packages from other publishers. Because their certificate won’t match the authors, so even if you trust the publisher, you only trust them for their apps.
- Plug and play solution, the only time you need to configure something is when a certificate changes or you trust a new publisher. Nothing else needs to be configured. - Adding a new dependency is as easy as putting it into your app.json.
- Code sign certificates usually have a longer lifetime. Additionally, you can be warned to add the publisher’s new certificate to your trusted dictionary. And there are ways to also automate this process.
Microsoft Application dependency
For an automatic resolution, an accurate dependency graph is key. All the actual dependency should be present in the manifest. This allows to resolve packages without the need to extract this information from the app package first, eliminating the need for a docker container when handling runtime apps.
One information that’s missing in the current proposal #261, is the reference to the Microsoft Application and its version.
In early versions this was reflected just as a regular dependency. Even if this is now expressed differently, the dependency is still there and will need to be added to the manifest. Even if there might be an arguable need for this right now - In my opinion it’s a must for accurate resolution – its better to add it anyway. It can be ignored if it’s not needed. But if a use case is discovered that depends on it, all previously created packages will break.
Lock file
In general BC specifies all version as minimum version. Which in means resolution should always locate the latest package that satisfy this minimum version. Since the dependency are not contained in the binary, a developer needs to account for running against various dependencies. – Not all SaaS environments will have the same dependency versions installed. The only guarantee a developer has is that the minimum requirements are satisfied.
But in some cases, it might be desirable to specify exact versions when resolving dependencies. Other package managers use a look file which specifies a different version constraint. The same concept can be adapted here as well.
An easy format would be use the following json
{
"<appid1>": "<nuget Version constraint>",
"<appid2>": "<nuget Version constraint>",
"<appid3>": "<nuget Version constraint>",
}
On this: It's important to talk about the reason why this is dangerous and why this danger is unique to business central.
For internal use, we can safely search for a package with an appid, as this is something we would have added ourselves. If we do the same on Nuget.org - that package may be published by somebody else. I can publish a package with somebody elses appid today and in 3 months from now, when somebody adds a dependency, they download my package.
This is why we (for external dependencies) cannot rely on automatic resolutions, but must have specific package dependency insertion.
For other products, you also search for Newtonsoft and find the right nuget package and add a reference to that. The Newtonsoft package is not just one DLL - it is a package and you can now use that. This is what I want to mimic. Add a reference to a package name, which you receive from a partner. This package is then known to be published by that partner and can never be overridden by somebody else (packages on NuGet.org can be unlisted, but the name can never be reused).
This is why we (for external dependencies) cannot rely on automatic resolutions, but must have specific package dependency insertion.
I think it is possible to use an automatic process. That what I described in the following part.
But there is a technical solution to this problem as well. Every partner already signs the apps that they create. Using this we can create a dictionary of trusted publisher name and public keys, from the code sign certificates.
When resolve a dependency you need to do the following steps.
Requirements: Collect the trusted dictionary.
- Locate dependency in the feeds.
- Download the dependency.
- Get public keys for the vendor you specified in you app.json
- Verify the downloaded dependency to be signed with a private key that correspond to one of the public keys.
- Repeat steps 2-6 until you hit an unverified package or have no more dependency to resolve.
- Use apps as needed.
If you verify that the app was signed by a publisher you trust. The attacker can only trick you in executing their code if they have the code signing certificate from the publisher - If they have that, you are done anyway.
For other products, you also search for Newtonsoft and find the right nuget package and add a reference to that.
This is what I hinted at here
You would only specify a dependency after it was deemed safe and reliable by a human. The foundation for that assessment is what’s in the public feed.
The Newtonsoft package is not just one DLL - it is a package and you can now use that.
But dependencies in business central are defined by app (which in your example is equivalent to a dll) And I think this is what should be located in the feed at least for automatic resolution. There is no additional configuration that needs to be maintained just the information you already need. This makes it effortless.
Add a reference to a package name, which you receive from a partner.
In my example the publisher would provide you with the public key. That way you know that you can trust all apps that were signed by that certificate. (At least if you were expecting the package to be from them)
Just replied to your email as well.
There is no way we (Microsoft) will be able to get a feature through a security review where we are downloading apps from a public site based on appids - whether they are signed or not. That is not a battle I am prepared to take AND I do not think that is the right way to go (even though it might be possible)
Also, I don't think the dependency description you have in app.json is sufficient to describe exactly which version of the app you want. It only describes a minimum version.
NuGet package versioning has a better versioning schema, where you can describe min and max versions of the NuGet package that's why I think you should describe what app you depend on in app.json and you describe what solutions you depend on in your repository (potentially incl. tokens etc).
Partners can decide to publish their apps as one app per nuget package with dependencies between packages, that is fully supported by the described model, but I still think you need to specify the starting point in settings - and not just magically get it from appids in app.json for external dependencies..,
Okay I got It thank you.
Maybe you can consider the following:
- Allow multiple feeds to be queried in sequence - maybe I got two internal feeds.
- Add the Microsoft Application as a dependency in the manifest.
- Resolve the dependency recursively according to the version ranges
- Provide a hook to validate a package after download.
And maybe one day you are prepared for that security review or maybe not...
That all sounds like some good implementation suggestions - and yes, maybe one day... :-)
@jonaswre - please re-read the nuget description. You will find that I now have implemented trusted feeds in a local BcContainerHelper version and that all packages now contains one app and all dependencies to other apps (including all microsoft packages),
Not completely done yet - but getting there.
@freddydk I like the changes you made. Specially the trusted feeds will be very usefull.
I have to questions/suggestions. Firstly I think patterns are a good short term solution to ensure only trused apps are downloaded from a feed. But I think for a long term solution the code signing should be used to ensure the apps are signed by the developer. I think this would ensure AL-Go to be secure by default.
Secondly the dependency resolution is still always resolving the latest dependency right? I can't resolve for example all apps thats are compatible with Application version 22.0?
The patterns are really from this: https://learn.microsoft.com/en-us/nuget/nuget-org/id-prefix-reservation#id-prefix-reservation-criteria - stating that you trust nuget.org / microsoft.* is safe.
Beside Microsoft - other partners can register their prefix and/or have their own feeds which also are safe.
The code signing could also be added to the mix, stating that I only want code signed packages and/or only want code signed packages from a specific vendor - I don't think we would sign microsoft packages though, so it should be "settable" on the feed. IMO - registered prefixes is as safe as code signing.
Correct - currently I still get latest, but it is on my list to be able to get apps compatible with a prior version.
The patterns are really from this: https://learn.microsoft.com/en-us/nuget/nuget-org/id-prefix-reservation#id-prefix-reservation-criteria - stating that you trust nuget.org / microsoft.* is safe. Beside Microsoft - other partners can register their prefix and/or have their own feeds which also are safe.
Yes you are right, if the source guarantees the prefix is reserved for a particular owner its pretty secure. Whats the default value for pattern? I think an entry without a pattern shouldn't resolve anything. (Secure by default).
IMO - registered prefixes is as safe as code signing.
Code signing should be more secure, since it doesn't depend on trusting any source. As long as you get the public key in a trusted manner, you can verify any downloaded app against your trusted publishers. If users don't copy and paste pattern: ['*'], pattern are pretty safe.
Correct - currently I still get latest, but it is on my list to be able to get apps compatible with a prior version. Thats good. But I think better runtime generation is more important.
There is no default pattern when adding a trusted feed, people will have to add that.
On the code signing...
- I guess we should register trusted hash'es on every feed and state on f.ex. nuget.org we trust packages signed by A, B or C.
- do you have an example of a signed package?
@jonaswre - as you will find - the functionality in BcContainerHelper has been refactored quite a bit and there is now also support for fingerprints in trusted feeds, meaning that you can select to trust only packages signed by a specific certificate. Note that all packages on nuget.org are signed by a repository certificate - you should not trust that - you should only trust packages signed by the author.
The naming of GitHub packages has also changed to become publisher.name.appid (namespace like), but resolution is done solely on appid.
Also when using functions like Publish-BcNuGetPackageToContainer - it will find the package matching the applications installed in the container (if your container is 22.0 and the latest app package requires 23.0 - it will look for a version, which requires what you have.
Many changes, but GitHub Packages functionality should still work as before in your repositories.
I will document everything over the next days.