matrix-docker-ansible-deploy
matrix-docker-ansible-deploy copied to clipboard
spantaleev
matrix-goofys: error related to fusermount
The service to enable and run goofys for filesystem fails due to the error related to fusermount when you restart matrix-goofys:
Jun 05 23:20:05 ik1-432-48064 systemd[1]: Starting Matrix Goofys media store...
Jun 05 23:20:05 ik1-432-48064 matrix-goofys[13842]: Error response from daemon: Cannot kill container: matrix-goofys.service: No such container: matrix-goofys.service
Jun 05 23:20:05 ik1-432-48064 matrix-goofys[13852]: Error: No such container: matrix-goofys.service
Jun 05 23:20:05 ik1-432-48064 systemd[1]: Started Matrix Goofys media store.
Jun 05 23:20:05 ik1-432-48064 matrix-goofys[13861]: 2021/06/05 14:20:05.982870 main.FATAL Mounting file system: Mount: mount: running fusermount: exit status 1
Jun 05 23:20:05 ik1-432-48064 matrix-goofys[13861]: stderr:
Jun 05 23:20:05 ik1-432-48064 matrix-goofys[13861]: fusermount: mountpoint is not empty
Jun 05 23:20:05 ik1-432-48064 matrix-goofys[13861]: fusermount: if you are sure this is safe, use the 'nonempty' mount option
Jun 05 23:20:06 ik1-432-48064 systemd[1]: matrix-goofys.service: Main process exited, code=exited, status=1/FAILURE
Jun 05 23:20:06 ik1-432-48064 systemd[1]: matrix-goofys.service: Failed with result 'exit-code'.
Jun 05 23:20:11 ik1-432-48064 systemd[1]: matrix-goofys.service: Service RestartSec=5s expired, scheduling restart.
Jun 05 23:20:11 ik1-432-48064 systemd[1]: matrix-goofys.service: Scheduled restart job, restart counter is at 338.
Restarting the server did not fix the error, I am wondering how it should be possible to troubleshoot and fix it to make matrix-goofys run successfully. Thanks!
$ uname -v
#1 SMP Debian 4.19.181-1 (2021-03-19)
The setup is based on 8cf3bdb7720d7b61fa32bc7e1290439c533fbe3a.
My bad, the issue is that the folder has been recreated at /matrix/synapse/storage/media-store somehow and the mount point was actually not empty as it says. Removing the folder manually and running ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start fixed the issue.
I would like to suggest to add something as a task to check whether the folder was empty and prompt the error if not.
Closing for now.
The issue keeps happening when you run ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start again.
For me it was fixed temporarily by removing /matrix/synapse/storage/media-store/remote_content manually to clear the content of the folder only after running that command.
sudo journalctl -fu matrix-goofys returns:
Jun 08 17:41:35 ik1-432-48064 systemd[1]: Starting Matrix Goofys media store...
Jun 08 17:41:36 ik1-432-48064 matrix-goofys[26459]: Error response from daemon: Cannot kill container: matrix-goofys.service: No such container: matrix-goofys.service
Jun 08 17:41:36 ik1-432-48064 matrix-goofys[26467]: Error: No such container: matrix-goofys.service
Jun 08 17:41:36 ik1-432-48064 systemd[1]: Started Matrix Goofys media store.
Jun 08 17:41:36 ik1-432-48064 matrix-goofys[26476]: 2021/06/08 08:41:36.583182 main.FATAL Mounting file system: Mount: mount: running fusermount: exit status 1
Jun 08 17:41:36 ik1-432-48064 matrix-goofys[26476]: stderr:
Jun 08 17:41:36 ik1-432-48064 matrix-goofys[26476]: fusermount: mountpoint is not empty
Jun 08 17:41:36 ik1-432-48064 matrix-goofys[26476]: fusermount: if you are sure this is safe, use the 'nonempty' mount option
Jun 08 17:41:36 ik1-432-48064 systemd[1]: matrix-goofys.service: Main process exited, code=exited, status=1/FAILURE
Jun 08 17:41:36 ik1-432-48064 systemd[1]: matrix-goofys.service: Failed with result 'exit-code'.
Jun 08 17:41:41 ik1-432-48064 systemd[1]: matrix-goofys.service: Service RestartSec=5s expired, scheduling restart.
Jun 08 17:41:41 ik1-432-48064 systemd[1]: matrix-goofys.service: Scheduled restart job, restart counter is at 111.
Jun 08 17:41:41 ik1-432-48064 systemd[1]: Stopped Matrix Goofys media store.
Jun 08 17:41:41 ik1-432-48064 systemd[1]: Starting Matrix Goofys media store...
Jun 08 17:41:41 ik1-432-48064 matrix-goofys[26571]: Error response from daemon: Cannot kill container: matrix-goofys.service: No such container: matrix-goofys.service
Jun 08 17:41:41 ik1-432-48064 matrix-goofys[26579]: Error: No such container: matrix-goofys.service
Jun 08 17:41:41 ik1-432-48064 systemd[1]: Started Matrix Goofys media store.
Jun 08 17:41:42 ik1-432-48064 matrix-goofys[26589]: 2021/06/08 08:41:42.434361 main.INFO File system has been successfully mounted.
The media-store folder was mounted as the last line says after removing the content of the folder on that timing.
The issue has not happened until the last month. I did not look into what has changed but the default setting might have been changed somewhere I guess.
The same issue started happening to me. I am able to start service by executing: rm -fr /matrix/synapse/storage/media-store/* but it re-appears once I run ansible-playbook -i inventory/hosts setup.yml --tags=setup-all and then ansible-playbook -i inventory/hosts setup.yml --tags=start (as described in https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/maintenance-upgrading-services.md#upgrading-the-matrix-services)
Yeah, this happens to me every time I have to reboot the server. I don't know what service write in the media folder making the goofy container to fail because there is already files in there, so I have to delete the content and restart the goofy container. But in the middle some media can be generated.
