Upgrade from 10.8.1 to 10.9 failed: feedback

[Biboba]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

Module Version

• 3.2.0

Affected Resource(s)

• PortalPostUpgradeV2

Configuration Files

# Copy-paste your DSC JSON configurations here - for large configs,
# please use a service like Dropbox and share a link to the ZIP file.

Expected Behavior

Upgrade from 10.8.1 to 10.9 should be sucessful.

Actual Behavior

It failed with the following error:

09-May-21 11:50:04 AM: PowerShell DSC resource ArcGIS_PortalUpgrade  failed to execute Set-TargetResource functionality with error message: [ERROR]:- "{\"status\":\"success with warnings\",\"messages\":[\"The index is not in sync. Perform a full reindex in the ArcGIS Portal Directory to ensure the portal is functioning correctly.\"],\"recheckAfterSeconds\":20}" 
09-May-21 11:50:04 AM: The SendConfigurationApply function did not succeed.

And here is a the log file associated:

09-May-21 11:46:23 AM: [GISPORTAL01]:                            [[ArcGIS_PortalUpgrade]PortalUpgrade] POST https://gisportal01.company.com:7443/arcgis/portaladmin/postUpgrade with 185-byte payload
09-May-21 11:50:04 AM: [GISPORTAL01]:                            [[ArcGIS_PortalUpgrade]PortalUpgrade] received -1-byte response of content type application/json;charset=UTF-8
09-May-21 11:50:04 AM: [GISPORTAL01]:                            [[ArcGIS_PortalUpgrade]PortalUpgrade] Response:- {"status":"success with warnings","messages":["The index is not in sync. Perform a full reindex in the ArcGIS Portal Directory to ensure the portal is functioning correctly."],"recheckAfterSeconds":20}
09-May-21 11:50:04 AM: [GISPORTAL01]:                            [[ArcGIS_PortalUpgrade]PortalUpgrade] Response received from post upgrade step {"status":"success with warnings","messages":["The index is not in sync. Perform a full reindex in the ArcGIS Portal Directory to ensure the portal is functioning correctly."],"recheckAfterSeconds":20}

I believe it failed because ArcGIS Powershell DSC is expecting a simple "success": https://github.com/Esri/arcgis-powershell-dsc/blob/78e488fc0aac6f8f4cdf574e1bdb50a7d39ed350/Modules/ArcGIS/DSCResources/ArcGIS_PortalUpgrade/ArcGIS_PortalUpgrade.psm1#L113

Then I completed a full reindex because the search "Count in store" was not synched with "Count in index". It did not solve the issue because they were still not in synched after the reindex.

Comparing with another upgrade I did on a dev server, I saw that the upgrade was pretty much done as you are only doing afterward a synch of living atlas. So I re-run the DSC config, it jumped the Portal upgrade step and finished successfully. Good news. Then I made a webgisdr backup but got the following warning:

During the export operation, some items could not be exported because they appear to be missing from the content directory. Check the portal logs and review the Why do I receive an error about a missing item when creating a backup of my portal? section in the Common Problems and Solutions documentation for more information.

Checking the Portal logs, 14 items could not be exported:

<Msg time="2021-05-09T22:12:45,859" type="WARNING" code="209129" source="Portal Admin" process="89600" thread="1" methodName="" machine="GISPORTAL01.COMPANY.COM" user="" elapsed="" requestID="">During the export operation, some items could not be exported because they appear to be missing from the content directory. Review the information under the "Why do I receive an error about a missing item when creating a backup of my portal?" section in the Common Problems and Solutions documentation to resolve the problem. The item ids for the items with the problem are: ba99a4a4f5ce48debbeca6713e051f1e, 7da16f48c81f448fa972d4a52fdc1e4e, 1462b2f8a2944a25ba04a81739202dba, 57f358c3c13e425cbc6c0a854bb3cdeb, 53137233a760432bb07c417eb3d758b8, b02a1a4bacd142abad5822bced7787e1, f59235b3e9714665a938f1089eed843b, c8c33e7a32b24f84adfea843d20eb2bf, 070ce5f4390c4be4b077ab88820052a7, 9c54652ce7b3472ea33751781ab3aade, 81c5a9f2a2704d54a49042a44eefa5d3, 03dd7806c56449368b232a72ba5e9e38, 8f7fe2c517084157a499b84f48fc0fa1, 5a477abfbe064b86b49319f6a9a2c4a9.</Msg>

Searching for those 14 items, I found out on the sharing API that all belong to "esri_livingatlas". I took a look at your script and manually did the 'Invoke-UpgradeLivingAtlas' on the Portal admin API: https://github.com/Esri/arcgis-powershell-dsc/blob/78e488fc0aac6f8f4cdf574e1bdb50a7d39ed350/Modules/ArcGIS/DSCResources/ArcGIS_PortalUpgrade/ArcGIS_PortalUpgrade.psm1#L357-L380

After that, I did another full reindex, and afterward, index was in sync once again. Running another webgisdr backup, I still get the same warning. I opened a ticket to support regarding that part but I am not sure it is good to make the DSC upgrade config fails if it's succeed with warning specially knowing that the following step after the upgrade, that is to say, updating living atlas content, and running Reindex fix the index not in synch.

Steps to Reproduce

Upgrading from 10.8.1 to 10.9:

Invoke-ArcGISConfiguration -ConfigurationParametersFile C:\tmp\DeploymentUpgrade.json -Credential (Get-Credential) -Mode Upgrade 

Important Factoids

I think living atlas content was disabled on my Portal for ArcGIS and now it seems enabled. I am currently rebuilding the old site in 10.8.1 in order to check if that was the case and if it could be somehow related.

Thanks !