[BUG] deploying a package which is currently deployed in a different version is not handled correctly

[line-o]

Describe the bug

Given a XAR package that is currently installed in version A. Uploading a XAR with version B and calling repo:install-and-deploy-from-db() creates a zombie package, which is almost impossible to remove once installed.

2020-11-01 19:24:01,776 [qtp157004450-23] INFO  (Deployment.java [installAndDeploy]:282) - Deploying package http://myapp/
2020-11-01 19:26:57,715 [qtp157004450-23] INFO  (Deployment.java [installAndDeploy]:269) - Installing package /db/system/repo/myapp-1.0.1.xar
2020-11-01 19:26:57,726 [qtp157004450-23] INFO  (FileSystemStorage.java [getContenDir]:229) - None of content dirs exist: '/Users/jll/customers/exist/develop/exist-distribution/target/exist-distribution-5.3.0-SNAPSHOT-dir/etc/../data/expathrepo/myapp-1.0.1/content' and '/Users/jll/customers/exist/develop/exist-distribution/target/exist-distribution-5.3.0-SNAPSHOT-dir/etc/../data/expathrepo/myapp-1.0.1/myapp'
2020-11-01 19:26:57,746 [qtp157004450-23] INFO  (Deployment.java [installAndDeploy]:282) - Deploying package http://myapp/
2020-11-01 19:30:41,272 [qtp157004450-28] INFO  (Deployment.java [installAndDeploy]:193) - Application package http://myapp/ already installed. Skipping.
2020-11-01 19:30:58,745 [qtp157004450-28] INFO  (Deployment.java [installAndDeploy]:269) - Installing package /db/system/repo/myapp-1.0.0.xar
2020-11-01 19:30:58,759 [qtp157004450-28] INFO  (FileSystemStorage.java [getContenDir]:229) - None of content dirs exist: '/Users/jll/customers/exist/develop/exist-distribution/target/exist-distribution-5.3.0-SNAPSHOT-dir/etc/../data/expathrepo/myapp-1.0.0/content' and '/Users/jll/customers/exist/develop/exist-distribution/target/exist-distribution-5.3.0-SNAPSHOT-dir/etc/../data/expathrepo/myapp-1.0.0/myapp'
2020-11-01 19:30:58,783 [qtp157004450-28] INFO  (Deployment.java [installAndDeploy]:282) - Deploying package http://myapp/

Removing it via package manager / dashboard reports success, but the application is still listed and no Error is reported in neither expath-repo.log nor exist.log.

The calls to repo:undeploy("http://myapp/") and repo:remove("http://myapp/") act as if everything was normal reporting result="ok" and true(). It will still be listed in the result of repo:list(), however.

Updates to a package only work if the version number is bumped. It is also not possible to downgrade/ rollback such an application. Installing a previous version of it will only reinstate the version with the highest version number.

In my tests restarting the database did not help to mitigate the issue. Also manually removing all versions of the XAR in /db/system/repo was not enough. Even repair:clean-all(), repair:repair() alone did not help.

The package repo seems to get corrupted, if repo:remove() and repo:undeploy() are not called each time before new package version is about to be installed.

data/expathrepo/.expath-pkg/package.xml now contains two or more entries for the same application, which seems to be the root cause.

update

Thanks to @joewiz asking what is logged to expath-repo.log. That led to the discovery of the actual issue.

Expected behavior

1. Report an error at startup, upon installation and removal, hinting at a duplicate entry in package.xml.
2. Reject installation when a previous version was not removed
3. add a way to repair the expath-repo (could be added to repair:repair)

To Reproduce

1. upload application XAR in version A
2. call repo:install-and-deploy-from-db("myapp-a.xar")
3. upload application XAR in version B
4. call repo:install-and-deploy-from-db("myapp-b.xar")
5. open package.xml, there are now two entries for myapp

Context (please always complete the following information):

• OS: macOS 10.15.7
• eXist-db version: 5.3.0-SNAPSHOT (2ac8622bee785e9e4b67ecb2bae40cffc3b9cd48)
• Java Version Java 1.8.0_252 [AdoptOpenJDK]

Additional context

• How is eXist-db installed? from source
• Any custom changes in e.g. conf.xml? none

[joewiz]

You can attach a xar if you rename its extension as zip.

[joewiz]

Do you observe any error messages accompanying these actions in exist.log or expath-repo.log?

[line-o]

expath-repo.log on installation of XAR

2020-11-01 19:17:41,825 [qtp157004450-43] INFO  (Deployment.java [installAndDeploy]:269) - Installing package /db/system/repo/myapp-1.0.0.xar
2020-11-01 19:17:41,843 [qtp157004450-43] INFO  (FileSystemStorage.java [getContenDir]:229) - None of content dirs exist: '/Users/jll/customers/exist/develop/exist-distribution/target/exist-distribution-5.3.0-SNAPSHOT-dir/etc/../data/expathrepo/myapp-1.0.0/content' and '/Users/jll/customers/exist/develop/exist-distribution/target/exist-distribution-5.3.0-SNAPSHOT-dir/etc/../data/expathrepo/myapp-1.0.0/myapp'
2020-11-01 19:17:42,711 [qtp157004450-43] INFO  (Deployment.java [installAndDeploy]:282) - Deploying package http://myapp/

[line-o]

@joewiz Thanks to you I noticed the log entries that hinted at problems in .expath-repo/package.xml. After re-testing it was clear that the package URI is not involved in any way. The XAR is also not important. I am unsure how to write a xquery testsuite that will upload a xar package, but will investigate if necessary.

[adamretter]

@line-o For an example take a look at PackageTriggerTest

[nverwer]

This issue is driving me crazy. Is there a way to actually remove a package, other than erasing the whole database and rebuilding it? Even a complicated manual procedure would be better than not being able to remove a package at all.

[Edit] To answer my own question: Go to the eXist data directory, and into the expathrepo sub-directory. Remove the directory of the zombie package. Then go into the .expath-pkg sub-directory. Edit both packages.txt and packages.xml, removing lines that mention the zombie package.

[line-o]

@nverwer short answer: use xst package install and xst package uninstall long answer: Make sure to call repo:remove($package-name) and repo:undeploy($package-name) before installing a new version of a package