Empty Encryption Key Error in Docspell Addon Authentication

[tiborrr]

With a little bit help from some AI I have managed to found the cause of this bug:

Description

When running Docspell with multiple JOEX nodes, authentication failures occur because the REST server node is missing from the node table in the database. This causes JOEX nodes to attempt token signing with an empty server secret, resulting in the following error:

java.lang.IllegalArgumentException: Empty key
 at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:107)
 at docspell.common.util.SignUtil$.getMac(SignUtil.scala:21)
 at docspell.common.util.SignUtil$.signString(SignUtil.scala:26)
...

Current Behavior

1. REST server starts with a configured docspell.server.auth.server-secret
2. At some point, the REST server node entry disappears from the node table
3. JOEX nodes can't find the REST server secret and default to an empty key
4. Authentication operations fail due to empty key in crypto operations

Database State

The node table only contains JOEX entries, missing the REST server:

"id","type","url","updated","created","not_found","server_secret"
docspell-joex1,joex,http://docspell-joex1:7878,2024-12-10 16:58:45.501,2024-12-10 09:58:38.303,0,
docspell-joex2,joex,http://docspell-joex2:7878,2024-12-10 16:58:45.511,2024-12-10 09:58:34.842,0,
docspell-joex3,joex,http://docspell-joex3:7878,2024-12-10 16:58:45.523,2024-12-10 09:58:39.408,0,

Expected Behavior

REST server should maintain its registration in the node table

Potential Investigation Points

1. REST server node registration process
2. Node cleanup/maintenance processes that might incorrectly remove the REST server entry
3. Error handling when server secret is missing (currently silently falls back to empty key)

Environment

• Multiple JOEX nodes in production setup
• REST server configured with server-secret
• Database shows only JOEX nodes, missing REST server entry

Additional Notes

This could be a race condition or timing issue where the REST server entry is being removed unexpectedly. The error handling could also be improved to fail fast with a clear error message when no server secret is available, rather than proceeding with an empty key.

[tiborrr]

@eikek I know you are very busy. But it would be great to have your vision on this issue. This is the main reason for my Docspell instance not running reliably. Do you know why the rest server might be unregistering?

[eikek]

Hi @tiborrr sorry this one got lost on my side, thanks for pinging.

I haven't had time to investigate. It is very strange that the rest-server entry is removed, this should only happen once it shuts down.

Do I understand correctly: this is about an addon that likes to run dsc commands which should be automatically setup to run commands against the restserver? I think the relevant code is here. It looks up some restserver and as you said falls back to an empty secret. In any case, this should error out with a proper message.

But the main problem is, that the enrty is missing in the table. I think one clue is that there are multiple joex instances. Maybe some query is messed up in this case. I can't see this issue on my instance, where I only have one joex. Needs more investigation. :-|

[tiborrr]

My custom (pure) addon does only one thing and that is update a custom field.

Do you know at what point in the code the restserver can get removed from the node table?

[eikek]

Do you know at what point in the code the restserver can get removed from the node table?

It is setup so that on startup the row is inserted and removed when the server shuts down:

https://github.com/eikek/docspell/blob/091bc22ac422b48f4314dc19fd76e297775aaaaf/modules/restserver/src/main/scala/docspell/restserver/RestServer.scala#L102

I can't see any other use of the delete or unregister call.

[tiborrr]

@eikek I have run the server now for a few more months now just with just one Joex instance. But I still see unregistering of the restserver happening. I think I will add a log to that line, because something fishy is going on.