microsoft
Pin module dependencies using version numbers when importing modules
We have various issues reported around Microsoft365DSC requiring specific versions of modules; and to enforce a module version, Uninstall-M365DSCOutdatedDependencies currently removes newer versions of a module as well as older:
- #5920
- #6160
- #6139
- #6136
While it's clear that we need to enforce module versions during Microsoft365DSC execution (because of underlying breaking changes between versions of the modules), I'm not sure why we aren't using Import-Module -FullyQualifiedName or -RequiredVersion and Import-DSCResource -ModuleVersion to 'pin' each version of Microsoft365DSC's usage of each individual module to the currently supported version?
If this was happening, surely Uninstall-M365DSCOutdatedDependencies could be updated to only remove older versions of a module - and people could keep newer versions installed on their devices, which would be helpful (see above issues?)
Is there a reason this wouldn't work?
(We already have the relevant information here in a format which is not far off from being usable by the '-FullyQualifiedName' parameter of Import-Module, and splatted to Import-DSCResource): https://github.com/microsoft/Microsoft365DSC/blob/Dev/Modules/Microsoft365DSC/Dependencies/Manifest.psd1)
I like that idea! Will check how we can do that during startup of M365DSC.
@FabienTschanz Fantastic! Looking through other PowerShell docs it looks like the official way to do this is to define RequiredModules in the module manifest. I think this would also allow Install-Module to automatically download the relevant versions from the PowerShell Gallery. I don't know why we don't do this already - perhaps two possible reasons?
- To work around .Net assembly conflicts? (loading the same module with two versions of e.g. Microsoft.Identity.Client in the 'wrong' order often causes issues under .Net Framework / Windows PowerShell)?
- We don't want the PowerShell gallery auto-behaviour for some reason?
But at face value, this does exactly what we want without having to update every Import-Module call.
@ykuijs @NikCharlebois Maybe you can shed some light on the RequiredModules vs. custom implementation discussion. I'll certainly evaluate both ways and come up with a proposal on what might work better.
Feedback from @ykuijs why it's the way it currently is:
A few years back we used the RequiredModules feature in the module manifest. This however had the downside that installing M365DSC also installed all required modules, which could take a long time (sometimes more than 10 minutes). With the increase of the number of Graph modules, it was expected that this time would only increase even more.
However, there are situations where you do not need all required modules. For example when compiling a configuration into a MOF file. In that instance, you only need the M365DSC module. Only when you deploy a configuration, you need the required modules. Adding the modules to the RequiredModules in the manifest would add more than 15 minutes to a build phase (for example in the whitepaper solution).
By removing the required modules from the manifest and creating the cmdlet, you can define when to install the required modules and when not to install them.
Another issue we encountered was the fact that if we require v1.2.3 but also had v1.2.4 is installed, the code would always use the newest version. We have seen conflicts and issue because of this. That is why we created the Uninstall cmdlet, so we can make sure just the required version of the modules are installed.
We therefore shouldn't use RequiredModules but rather come up with another solution that loads only the specified version of the modules into the current scope and removes all others. This could be done at startup, PowerShell will keep the scope open as well as the LCM. Startup time with loading the modules could potentially increase by a bit.
That figures. Another downside of RequiredModules that I've found recently is that some modules (specifically Az.Accounts) have various .Net assemblies defined in the 'RequiredAssemblies' of their own module definitions, and if these are 'out of date' compared to what another PowerShell module requires e.g. MicrosoftGraph, you get all sorts of assembly clashes, e.g. https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/2148
These are hard enough to work around as it is (often involving running the various Connect-* cmdlets in a specific order, depending on which modules are currently 'ahead' in their version assemblies) and because Az.Accounts brings everything in the moment you 'touch' the module (unlike the rest, which only seem to import their assemblies once you run the Connect-* cmdlet), importing that module in particular into Microsoft365DSC as part of RequiredModules, would probably make things worse.
Apparently it's better in PowerShell 7+, but that's not an option at present for Microsoft365DSC, because it's got a Windows PowerShell 5.1 dependency, because of the LCM, yadda yadda.
Would love to see if Import-Module -FullyQualifiedName or Import-Module -RequiredVersion could be used to import a specific version though.
Will try it out over the weekend and give some feedback. Hope that works well enough. I still have an issue where connecting to Graph first and then to SPO leads to Get-PnPApp not returning anything... It would be solved in PS7 😕 Hope that if we first import the SPO module, the issue would be solved.
Quick update: I tested a couple of versions that pre-imported the modules during startup - This led to such a bad loading time that it is far from usable (we're talking about minutes until the export can start).
A more dynamic approach using lazy loading of modules was tested as well - There, I implemented a check during Confirm-M365DSCDependencies if the currently loaded modules have the same version as the ones in the module manifest. This wasn't too bad and it correctly unloaded / reloaded the modules if the version wasn't correct. But this also means that for the first resource, the module version could potentially be higher than what's in the manifest, leading to unexpected handlings of the commands, e.g. for number of properties, structure of the output etc.
A possible solution would be to have a RequiredModules section in the settings.json (preferably) or a #Requires statement for the module in the .psm1 file. This way, it would be possible to forcibly remove modules of other versions and use only what's written in the manifest.
Any other ideas or suggestions? I'm not really a big fan of extending all settings.json files with another property (manually or whatever). And if so, we should include other changes to it that would solve even more issues at the same time.
@FabienTschanz Is there any mileage in playing with $PSModuleAutoloadingPreference then using Import-Module with pinned versions, before running each module?
@Borgquite That was one of my thoughts as well. Downside is that we currently don't know which modules are required on a per resource base, and importing all of the modules takes ages to complete and blows up the memory used for the process immensely. If we work on a list of required modules per resource, we could leverage that feature without issues though. As I said, it takes ages to complete, but once done, it works without any issues, even when n number of versions of a module are installed.
@FabienTschanz Maybe I'm missing something, but can't we work out which modules are required per resource by checking which New-M365DSCConnection Workloads are called? (And in my imagination, can't we actually perform the module check/load as part of that function?)
(If not then we should be able to work out what modules each resource requires fairly automagically I would have thought - if it calls a *-Mg cmdlet, it needs Microsoft.Graph, if it calls *-Az, it needs Az, etc etc. I guess EXO may be more challenging although we could force the issue one-off by using -Prefix 'EX' or similar within Connect-ExchangeOnline and updating all the exisiting resources to use that).
@Borgquite I'd agree to about 50%. The workload doesn't inherently tell us, which cmdlets and modules are required. Some SPO resources e.g. require additional Microsoft.Graph modules, whereas Intune can include X numbers of Graph modules, and the AAD ones as well. It's the number of modules required that's just so huge.
As an example: If we want to load the Intune workload and we're going with "that requires the Microsoft.Graph" modules, we're talking about 24 modules which need to be loaded. Same for AAD. The import of those 24 modules takes even on fast computers multiple minutes. I don't want to imagine the loading time on a runner with limited compute capacity.
You might have seen the delay when starting a LCM configuration run using Start-DscConfiguration that the first resource always takes quite long. On my desktop it takes about 30 seconds, on my laptop just shy of a minute. This is because of only a couple of Graph modules that need to be loaded, and here we only have few loaded (about 3-5). Now bump that up to 24 modules and we're talking about a 5 minute delay just in starting the first run for a single resource 😓
I'd rather like to have a solution where we can specify which module is required by a resource and if it's not yet loaded, we import them beforehand using version pinning. I think I'm able to write a parser which can tell me, for every resource, which modules are required, and automatically update the settings.json (and also align the format of it so that all files look the same - currently it's disgustingly bad sometimes).
@FabienTschanz Ah, yes I'd completely forgotten that the Microsoft.Graph and Az modules come in many parts - I guess this is where lazy module loading really does make you lazy!
I like the idea of a per-module check and just-in-time loading, that sounds like a good way ahead.
@FabienTschanz @NikCharlebois Great that this is now imported? Did the pull request implement this in its entirety?
(Can we have newer versions of the Graph module installed now, and do Test-M365DSCDependenciesForNewVersion, Uninstall-M365DSCOutdatedDependencies and Update-M365DSCDependencies now leave newer modules alone?)
The PR was reverted (not because the PR itself was faulty, but it showed an issue of MSCloudLoginAssistant). I've reached out to @NikCharlebois to update the dependency with my fix for it and then revert the revert of the implementation. Afterwards, I'll check again.
@Borgquite The behaviour is the following:
Test-M365DSCDependenciesForNewVersionwill still report modules from the gallery which are of a higher version than we have in our manifest. No changes thereUninstall-M365DSCOutdatedDependencieswill still uninstall outdated dependencies. That's the whole purpose of the cmdlet.Update-M365DSCDependenciesdid not receive any changes as well. If the module with manifest version X is not found, it will be installed.Update-M365DSCModulegot a new parameter-NoUninstallwhich means what it says: It won't uninstall old dependencies. This is not the default behaviour because it's not the intention to make it behave differently (folks who always used it to uninstall dependencies will probably appreciate it), but all the "power users" who want the modules to persist can now use this parameter.
@FabienTschanz Great! Does the documentation for Update-M365DSCModule get auto-updated?
https://microsoft365dsc.com/user-guide/cmdlets/Update-M365DSCModule/
Well, previously it did. Now, I'm not sure anymore. Will have to check, maybe we need to add a separate pipeline which opens a PR to update the documentation. Previously it was run when a merge to main happened, but since push protection was enabled from Microsoft on the repository, that doesn't work anymore.
@Borgquite Now it's finally going to be available (again) 👀 Had to fix a couple of issues with the LCM and MSCloudLoginAssistant, and then the PowerShell Gallery didn't allow to upload any new version (internal issues). But now we should be good.