Support enabling IPM instance-wide

[isc-tleavitt]

This is done via new options in the existing zpm "enable" command that add package mappings.

We looked into using %ALL but ultimately that doesn't work, because namespace-specific mappings aren't allowed to override %ALL mappings (!?).

Fixes #470

[gjsjohnmurray]

namespace-specific mappings aren't allowed to override %ALL mappings (!?).

This always struck me as an unfortunate design choice. Too late to get it changed now, but maybe a second pseudo-namespace %DEFAULT could be added whose mappings defer to namespace specific ones.

[isc-tleavitt]

namespace-specific mappings aren't allowed to override %ALL mappings (!?).

This always struck me as an unfortunate design choice. Too late to get it changed now, but maybe a second pseudo-namespace %DEFAULT could be added whose mappings defer to namespace specific ones.

@gjsjohnmurray thanks for chiming in! I like the idea and will float it, but we won't be able to make it a prerequisite for this work.

[isc-shuliu]

Not sure why the CIs are failing.

Tried to run the tests locally on 9b19985 it still fails although CIs passed in May ...

[isc-shuliu]

It appears the CI failure stems from this block https://github.com/intersystems/ipm/blob/bf8ed87be9f4abab369e93b217d5e42dc5d23bc6/src/cls/IPM/Lifecycle/Base.cls#L1069

[isc-tleavitt]

Next steps here for @isc-shuliu who's carrying this work forward:

• Revert the changes to the MapOnly installer XData block and associated method, as they're causing the CI issues and are generally problematic
• Reevaluate against criteria in https://github.com/intersystems/ipm/issues/470

[isc-shuliu]

CIs are failing specifically in I4H because migration was not successful (see log line 1362). This failure was introduced by changes in bf6e699 which attempts to perform data migration in all namespaces

Guess I shouldn't perform it in all namespaces according to this comment https://github.com/intersystems/ipm/blob/6a9dd0101cfadbb4d2a92bb660b0361ef32f12a9/src/cls/IPM/Utils/Migration.cls#L29 but I'm not sure why.

@isc-tleavitt Any insights?

[isc-shuliu]

This PR should now meet the requirement in https://github.com/intersystems/ipm/issues/470#issuecomment-2368203299 . Specifically,

It should be easy to say, after installing IPM, "I want this version of IPM to be used in all namespaces"

This can be done by mapping the current IPM globally

It should be easy to upgrade IPM instance-wide from any namespace to which the instance-wide installation is mapped (note: IPM installed version metadata needs to be tracked in the right DB!) - this might mean tracking the namespace through which IPM is managed, and only allowing upgrades through that namespace.

This is achieved by checking whether IPM in current NS is mapped if a user attempts to install/reinstall/uninstall IPM. If IPM is mapped from elsewhere, will abort and prompt the user to go the source namespace.

It should be easy, with or without an instance-wide installation of IPM, to install and run different IPM versions in different namespace.

This is achieved by adding the unmap command which allows a user to unmap IPM in give namespaces.

There should be a way to see which namespaces have IPM available, at which versions, and where it's mapped from in each namespace.

The zpm "version" command will show IPM versions from all namespaces, and if applicable, where it's mapped from.

Interactive use of the zpm command should direct the user properly if IPM is not available in the current context.

If IPM is not installed, will iterate through all namespaces and run zpm "version" in the first namespace found that has IPM installed.

After enabling of IPM in a namespace with 0.7.x (or earlier) installed, need to run the data migration process.

%IPM.Utils.Migration:MigrateZPMToIPM() will be called in each namespace

[isc-tleavitt]

CIs are failing specifically in I4H because migration was not successful (see log line 1362). This failure was introduced by changes in bf6e699 which attempts to perform data migration in all namespaces

Guess I shouldn't perform it in all namespaces according to this comment

https://github.com/intersystems/ipm/blob/6a9dd0101cfadbb4d2a92bb660b0361ef32f12a9/src/cls/IPM/Utils/Migration.cls#L29

but I'm not sure why. @isc-tleavitt Any insights?

The issue here is:

Migrating ZPM data to IPM... in namespace HSCUSTOM
Migrating repo registry... 
IPM data migration was not fully successful. You may be able to reinstall the impacted packages to correct the issues.
ERROR #5809: Object to Load not found, class '%IPM.Storage.Module', ID '1'

Migrating ZPM data to IPM... in namespace HSLIB
IPM data migration was not fully successful. You may be able to reinstall the impacted packages to correct the issues.
ERROR #5002: ObjectScript error: <PROTECT>SetMapSelectability+17^%SYSTEM.SQL.Util.1 ^rINDEXEXT("MS","%IPM_Storage","ModuleItem","IDKEY"),/usr/irissys/mgr/hslib/

Migrating ZPM data to IPM... in namespace HSSYS
Migrating repo registry... 
IPM data migration was not fully successful. You may be able to reinstall the impacted packages to correct the issues.
ERROR #5809: Object to Load not found, class '%IPM.Storage.Module', ID '1'

Migrating ZPM data to IPM... in namespace HSSYSLOCALTEMP
Migrating repo registry... 
IPM data migration was not fully successful. You may be able to reinstall the impacted packages to correct the issues.
ERROR #5809: Object to Load not found, class '%IPM.Storage.Module', ID '1'

Two classes of issues here:

• HSLIB: here we can't migrate because the HSLIB database is read-only. In this case, we want to just skip. (Should be able to detect this.)
• Others: likely related to presence of IPM in HSLIB itself but kind of hard to pinpoint with available data.

Ultimately, I think the issue is going to be that we should only try to migrate a namespace's data after it's mapped.

Data migration post-installation should migrate everywhere that is already mapped (mostly will be useful for future automatic data migrations); mapping should migrate the newly-mapped namespace. The automatic mapping post-installation on an environment with older ZPM will end up migrating each namespace as it's mapped. That covers all scenarios.

[isc-shuliu]

@isc-tleavitt I updated the migration logic to run only in namespaces where %IPM is mapped from the current namespace. This resolved the problem for HSLIB but still doesn't successfully migrate HSCUSTOM, HSSYS, and HSSYSLOCALTEMP.

In I4H, HSCUSTOM, HSSYS, and HSSYSLOCALTEMP did not have %IPM mapping initially. Upgrading from %ZPM to %IPM automatically adds %IPM mapping for them. However, data migration still fails in these 3 namespaces.

[isc-shuliu]

@isc-tleavitt I think I figured out why:

In ##class(%IPM.Utils.Migration).MigrateOneRepo(), we should check for

Set newObj = ##class(%IPM.Repo.Definition).%OpenId(newId,,.sc)

instead of

Set newObj = ##class(%IPM.Storage.Module).%OpenId(newId,,.sc)

This is likely an error due to copy-paste from MigrateOnePackageToIPM()

[isc-tleavitt]

@isc-shuliu goooood catch! Thank you!

[isc-tleavitt]

I can't approve this (since it was originally my PR), but I think it's in a good state; would appreciate an eye on this from @isc-kiyer next week.