ejabberd 17.1 - make uninstall doesn't remove all files

[cfabio]

What version of ejabberd are you using?

ejabberd 17.1

What operating system (version) are you using?

CentOS 7.3 x64 System packages are up to date and Erlang OTP version is 19.2

How did you install ejabberd (source, package, distribution)?

Compiled from source

What did not work as expected? Are there error messages in the log? What was the unexpected behavior? What was the expected result?

After upgrading from ejabberd 16.12 to ejabberd 17.1 clients cannot authenticate anymore and ejabberd's log is spammed with the following repeating messages:

2017-01-27 13:13:38.268 [info] <0.539.0>@ejabberd_listener:accept:333 (#Port<0.23711>) Accepted connection xxx.xxx.xxx.xxx:34332 -> xxx.xxx.xxx.xxx:5222
2017-01-27 13:13:38.385 [info] <0.960.0>@ejabberd_c2s:wait_for_sasl_response:824 ({socket_state,fast_tls,{tlssock,#Port<0.23711>,#Port<0.23712>},<0.959.0>}) Accepted authentication for user by undefined from xxx.xxx.xxx.xxx
2017-01-27 13:13:38.463 [error] <0.960.0> gen_fsm <0.960.0> in state wait_for_bind terminated with reason: no function clause matching xmpp_codec:encode({sm_failed,'item-not-found',undefined,<<"urn:xmpp:sm:3">>}, <<"jabber:client">>) line 5383
2017-01-27 13:13:38.463 [error] <0.960.0> CRASH REPORT Process <0.960.0> with 0 neighbours exited with reason: no function clause matching xmpp_codec:encode({sm_failed,'item-not-found',undefined,<<"urn:xmpp:sm:3">>}, <<"jabber:client">>) line 5383 in p1_fsm:terminate/8 line 760

Do you have any suggestion on how to resolve the issue?

[zinid]

You have outdated xmpp library. Please rebuild ejabberd after make distclean.

[cfabio]

I managed to install 17.1 correctly, thank you for pointing me somewhat in the right direction. make distclean was not the problem tho, when upgrading to a new version of ejabberd I always download the zip, extract the files and compile. Ejabberd installation procedure creates a multitude of directories in /usr/lib, when running make uninstall those directories are not deleted. To solve the issue I uninstalled ejabberd 16.12 with make uninstall, cleared manually /usr/lib of the outdated packages (e.g. xmpp-1.1.2, xmpp-1.1.4, xmpp-1.1.6, etc) and then proceeded to compile and install ejabberd 17.1. Is there a way to tell make uninstall to clear /usr/lib also or do I have to do it myself every time I upgrade ejabberd? I never had this kind of issue all the other times I did upgrade to a new version of ejabberd.

[weiss]

Personally, when compiling ejabberd from source, I tell ./configure to install everything below a dedicated directory (/home/xmpp/server or /opt/ejabberd or whatever). I keep the configuration, data, and log files outside this directory (by using an ejabberdctl wrapper that sets --config-dir, --logs, and --spool accordingly). That way, I can just rm -rf ejabberd's installation directory before upgrading.

[zinid]

Ejabberd installation procedure creates a multitude of directories in /usr/lib, when running make uninstall those directories are not deleted.

@weiss this probably should be fixed anyway, no?

[weiss]

@weiss this probably should be fixed anyway, no?

That would be nice of course :-) And yes, I can reproduce the problem.

[cfabio]

For now I have instructed the Ansible script I use to automate the upgrade procedure to clear /urs/lib of the unneeded libraries. Putting all the files in a single directory is probably a very bad idea and will surely require me to add some SELinux policies. I don't like it, it's bad practice for a multitude of reasons.

[weiss]

it's bad practice

I think most admins would consider make install to be a "bad practice" for deploying production servers. But this all depends on your local environment and workflows.

I don't like it

I can easily accept that :-)

[zinid]

I set this 17.02-beta milestone, will cope with this closer to release date.

[badlop]

By just reusing some of the code for installing, and switching it to remove, I was able to almost get uninstall running. But strangely it doesn't remove one file per dir, so the directories aren't effectively removed after all. Here it is, just in case anybody can take a look, until there's a better alternative that actually works:

From cc437eaeefabd809a8a4ad5db89bb4a317448ff5 Mon Sep 17 00:00:00 2001
From: Badlop <[email protected]>
Date: Wed, 22 Feb 2017 16:45:04 +0100
Subject: [PATCH] preliminary code to uninstall deps, it forgets one file per
 dir, so dirs aren't deleted

---
 Makefile.in | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)

diff --git a/Makefile.in b/Makefile.in
index 28c0422..853709d 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -1,5 +1,6 @@
 REBAR = @ESCRIPT@ rebar
 INSTALL = @INSTALL@
+UNINSTALL = /bin/rm -d -v
 SED = @SED@
 ERL = @ERL@
 
@@ -121,8 +122,6 @@ TO_DEST=$(foreach path,$(1),$(call JOIN_PATHS,$(call TO_DEST_SINGLE,$(subst /, ,
 FILTER_DIRS=$(foreach path,$(1),$(if $(wildcard $(path)/*),,$(path)))
 FILES_WILDCARD=$(call FILTER_DIRS,$(foreach w,$(1),$(wildcard $(w))))
 
-ifeq ($(MAKECMDGOALS),copy-files-sub)
-
 DEPS:=$(sort $(shell $(REBAR) list-deps|$(SED) -e '/^=/d;s/ .*//'))
 
 DEPS_FILES=$(call FILES_WILDCARD,$(foreach DEP,$(DEPS),deps/$(DEP)/ebin/*.beam deps/$(DEP)/ebin/*.app deps/$(DEP)/priv/* deps/$(DEP)/priv/lib/* deps/$(DEP)/priv/bin/* deps/$(DEP)/include/*.hrl deps/$(DEP)/lib/*/ebin/*.beam deps/$(DEP)/lib/*/ebin/*.app))
@@ -139,6 +138,8 @@ endef
 $(foreach DEP,$(DEPS),$(eval $(call DEP_VERSION_template,$(DEP),deps/$(DEP)/ebin/$(DEP).app)))
 $(eval $(call DEP_VERSION_template,ejabberd,ebin/ejabberd.app))
 
+ifeq ($(MAKECMDGOALS),copy-files-sub)
+
 define COPY_template
 $(call TO_DEST,$(1)): $(1) $(call TO_DEST,$(dir $(1))) ; $$(INSTALL) -m 644 $(1) $(call TO_DEST,$(1))
 endef
@@ -161,11 +162,39 @@ copy-files-sub2: $(call TO_DEST,$(DEPS_FILES) $(MAIN_FILES) priv/bin/captcha.sh
 
 endif
 
+ifeq ($(MAKECMDGOALS),uncopy-files-sub)
+
+define COPY_template
+$(call TO_DEST,$(1)): $(1) $(call TO_DEST,$(dir $(1))) ; $$(UNINSTALL) $(call TO_DEST,$(1))
+endef
+
+$(foreach file,$(DEPS_FILES_FILTERED) $(MAIN_FILES),$(eval $(call COPY_template,$(file))))
+
+$(sort $(call TO_DEST,$(MAIN_DIRS) $(DEPS_DIRS))):
+	$(UNINSTALL) $@
+
+$(call TO_DEST,deps/epam/priv/bin/epam): $(LIBDIR)/%: deps/epam/priv/bin/epam $(call TO_DEST,deps/epam/priv/bin/)
+	$(UNINSTALL) $(O_USER) $< $@
+
+$(call TO_DEST,priv/sql/lite.sql): sql/lite.sql $(call TO_DEST,priv/sql)
+	$(UNINSTALL) $< $@
+
+$(call TO_DEST,priv/bin/captcha.sh): tools/captcha.sh $(call TO_DEST,priv/bin)
+	$(UNINSTALL) $< $@
+
+uncopy-files-sub2: $(call TO_DEST,$(DEPS_FILES) $(MAIN_FILES) priv/bin/captcha.sh priv/sql/lite.sql)
+endif
+
 copy-files:
 	$(MAKE) copy-files-sub
 
 copy-files-sub: copy-files-sub2
 
+uncopy-files:
+	$(MAKE) uncopy-files-sub
+
+uncopy-files-sub: uncopy-files-sub2
+
 install: all copy-files
 	#
 	# Configuration files
@@ -231,7 +260,7 @@ install: all copy-files
 		|| echo "Documentation not included in sources"
 	$(INSTALL) -m 644 COPYING $(DOCDIR)
 
-uninstall: uninstall-binary
+uninstall: uninstall-binary uncopy-files
 
 uninstall-binary:
 	rm -f  $(SBINDIR)/ejabberdctl
-- 
2.7.4

[cromain]

I reproduce the "doesn't remove one file per dir" and would also add it does not remove the last file for a given extension. Example after 'make uninstall':

ls /tmp/ejabberd/lib/esip-1.0.11/ebin/ esip.app esip_dialog.beam

Also installed directories will always remains, as we won't rm $SBINDIR $DOCDIR ... (they can be system directory, such as /usr/local/bin) This patch is not acceptable, we have to find a better solution

[badlop]

Timeline:

• https://github.com/processone/ejabberd/commit/ed124e15e9e2705ad9d5cddae284b85efd90fe7f: make install was implemented
• https://github.com/processone/ejabberd/commit/bfb7583bb287e3e4ccabdf8b7e109702366b2971: make uninstall was added
• https://github.com/processone/ejabberd/commit/4d8f7706240a1603468968f47fc7b150b788d62f: The rebar2 build tool was introduced, but the libraries were simply copied
• https://github.com/processone/ejabberd/commit/70606667c60bfc3196defe86056a5eb77841dfd5: make install was updated to replicate the OTP directory structure, but uninstall was not updated accordingly

Now that it is possible to build an OTP release easily, there is no need to replicate its directory structure in our Makefile.in. This reduces the code in Makefile.in to install and to uninstall, and prevents problems like the one mentioned in this issue.

In this sense, ejabberd 24.02 adds new Makefile experimental targets: make install-rel and make uninstall-rel. If they don't provoke other problems, in the future they could replace the old install/uninstall implementation, which would solve this issue.