Taskd server silently fails to start (any way to get more debug info than debug.tls=3 ?)

[traycerb]

I had this issue trying to run taskserver on Docker container ( https://github.com/ogarcia/docker-taskd) on a Synology NAS. Initially, it worked fine, but on server restart, trying to use the same certificates, it would silently exit, and the debug wasn't revealing. Here is what happened when I tried launching the server directly from within docker using verbose logging:

/var/taskd/pki # taskd server --debug --debug.tls=3
s: INFO Client certificate will be verified.
s: 3 ASSERT: x509_ext.c[gnutls_subject_alt_names_get]:111
s: 3 ASSERT: x509.c[get_alt_name]:2012
s: 3 ASSERT: mpi.c[wrap_nettle_mpi_print]:60
s: 2 added 6 protocols, 29 ciphersuites, 19 sig algos and 10 groups into priority list
/var/taskd/pki # ps -leaf
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh
   71 root      0:00 ps -leaf
/var/taskd/pki #

taskd silently fails, or at least I can't understand why the process is no longer running.

As mentioned, when I first created the container from the image, and gave it the relevant environment variables to create new certificates, it worked fine, remotely and all. I copied the brand-new certificates to my client and created permanent volume for the docker image so I wouldn't have to copy the certificates to the client again, as was advised:

mounting your own /var/taskd data volume with existing certificates.

This worked great, and even remotely too, but I needed to update and restart my server (Synology NAS running it's own flavor of Linux), and upon restarting the Docker taskd server , nothing works. There are no errors, the taskd server starts and exits within a few seconds.

docker run -d --name=taskd -p 53589:53589 -v /srv/taskd:/var/taskd connectical/taskd

running docker ps -a shows taskd exits after starts (this was screenshotted later, which is why it says 57 minutes)

root@XXXXXXXXXX:/srv/taskd/pki# docker ps -a
CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                      PORTS                      NAMES
6c8905c369d7   connectical/taskd                "/app/taskd/run.sh"      57 minutes ago   Exited (0) 57 minutes ago                              taskd

I thought it might be a permissions thing so I opened a session using interactive mode:

sudo docker run -ti -p 53589:53589 -v /srv/taskd:/var/taskd connectical/taskd /bin/sh

Running the diagnostics, I don't see any issue:

/var/taskd/pki # taskd diagnostics --data /var/taskd

taskd 1.1.0
    Platform: Linux
    Hostname: 1583bff7e60b

Compiler
     Version: 12.2.1 20220924
        Caps: +stdc +stdc_hosted +200809 +LP64 +c8 +i32 +l64 +vp64 +time_t64
  Compliance: C++11

Build Features
       Built: Oct 14 2022 15:22:08
       CMake: 3.24.2
     libuuid: libuuid + uuid_unparse_lower
   libgnutls: 3.7.8
  Build type: None

Configuration
   TASKDDATA: /var/taskd
        root: /var/taskd (readable)
      config: /var/taskd/config (readable)
          CA: /var/taskd/pki/ca.cert.pem (readable)
 Certificate: /var/taskd/pki/server.cert.pem (readable)
         Key: /var/taskd/pki/server.key.pem (readable)
         CRL: /var/taskd/pki/server.crl.pem (readable)
         Log: /var/taskd/log/taskd.log (found)
    PID File: /var/taskd/taskd.pid (found)
      Server: XXXXXXXXXXXXX:53589
 Max Request: 1048576 bytes
     Ciphers:
       Trust: strict

The log provides no extra information:

2023-05-13 22:43:52 ==== taskd 1.1.0  ====
2023-05-13 22:43:52 Serving from /var/taskd
2023-05-13 22:43:52 Debug mode
2023-05-13 22:43:52 Using address XXXXXXXXXXXX
2023-05-13 22:43:52 Using port 53589
2023-05-13 22:43:52 Using family
2023-05-13 22:43:52 Queue size 10 requests
2023-05-13 22:43:52 Request size limit 1048576 bytes
2023-05-13 22:43:52 IP logging on
2023-05-13 22:43:52 CA          /var/taskd/pki/ca.cert.pem
2023-05-13 22:43:52 Certificate /var/taskd/pki/server.cert.pem
2023-05-13 22:43:52 Private Key /var/taskd/pki/server.key.pem
2023-05-13 22:43:52 CRL         /var/taskd/pki/server.crl.pem
2023-05-13 22:43:52 Server starting
2023-05-13 22:43:57 Try again

My config is pretty straightforward:

confirmation=1
extensions=/usr/libexec/taskd
ip.log=on
log=/var/taskd/log/taskd.log
pid.file=/var/taskd/taskd.pid
queue.size=10
request.limit=1048576
root=/var/taskd
server=XXXXXXXXXXXXXXX:53589
trust=strict
verbose=1
client.cert=/var/taskd/pki/client.cert.pem
client.key=/var/taskd/pki/client.key.pem
server.cert=/var/taskd/pki/server.cert.pem
server.key=/var/taskd/pki/server.key.pem
server.crl=/var/taskd/pki/server.crl.pem
ca.cert=/var/taskd/pki/ca.cert.pem
debug.tls=3

Any ideas? Can you replicate this? Strangely, removing the existing certificates and generated them again, the taskd server will work fine, but copying the certificates each time isn't practical. I'm not sure if it's a taskd thing or a Docker thing, but I'm at the limits of my ability to understand it.

Any ideas how to pry more debug information from taskd? More info about why it's failing would be helpful.

Originally posted by @traycerb in https://github.com/ogarcia/docker-taskd/issues/11#issuecomment-1546764399