Cronicle doesn't start on boot

[Fradd747]

Summary

Cronicle won't start on boot. I followed https://github.com/jhuckaby/Cronicle/blob/master/docs/CommandLine.md#server-startup and still doesn't work. When I reboot my machine cronicle status is 'not running'.

Steps to reproduce the problem

Setup server using https://github.com/jhuckaby/Cronicle/blob/master/docs/Setup.md#setup, then enable run on boot in https://github.com/jhuckaby/Cronicle/blob/master/docs/CommandLine.md#server-startup.

Your Setup

Basic server install, single machine, local disk db.

Operating system and version?

Ubuntu 22.04.4 LTS

0.9.53

Are you using a multi-server setup, or just a single server?

single server

Are you using the filesystem as back-end storage, or S3/Couchbase?

back-end storage

Can you reproduce the crash consistently?

yes

Log Excerpts

[jhuckaby]

Please check the Cronicle logs after a reboot, to see if systemd attempted to start it and failed, or didn't attempt it at all. Specifically this log: /opt/cronicle/logs/Cronicle.log

And also /opt/cronicle/logs/crash.log if it exists.

Thank you.

[Fradd747]

Thanks for quick reply. crash.log doesn't exist and Cronicle.log is empty.

[jhuckaby]

Hmm, I don't know what else to try. What does systemctl status cronicle output for you?

[Fradd747]

Thank you, systemctl actually helped. It says /opt/cronicle/bin/control.sh: 70: node: not found, so I created symlink to /usr/bin using sudo ln -sf $(nvm which 20.13.1) /usr/bin/node, reboot machine and now it works perfect.

[pixelwave]

I ran into the same issue. Fresh Debian install and fresh install of cronicle and it also did not boot on startup properly.

I tried your command sudo ln -sf $(nvm which 22.11.0) /usr/bin/node

(altered the nvm version number to the current one I have)

Now it works on boot! I am just wondering can things break again should I update NVM or cronicle?

[jhuckaby]

As long as node exists in one of the standard locations like /usr/bin/node you should be fine.

[ioqy]

@jhuckaby

Same problem here.

Fresh Ubuntu 24 server and followed the node.js installation instructions from https://nodejs.org/en/download/

# Download and install nvm:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Download and install Node.js:
nvm install 22

# Verify the Node.js version:
node -v # Should print "v22.12.0".
nvm current # Should print "v22.12.0".

# Verify npm version:
npm -v # Should print "10.9.0".

Node is installed under /root/.nvm/versions/node/v22.12.0/bin/node

[jhuckaby]

@ioqy Start on boot only works if node is installed under one of the standard Linux binary locations:

PATH=$PATH:/usr/bin:/bin:/usr/local/bin:/usr/sbin:/sbin:/usr/local/sbin

You can symlink it there using a command such as:

ln -snf /root/.nvm/versions/node/v22.12.0/bin/node /usr/bin/node

[ioqy]

Thanks for the hint.

While the installation mentions that node "must be installed to a standard location", it neither mentions how to do this or that the default installation method of node (recommended by the cronicle setup guide) doesn't do this.

So to me it looks like the default location of node is most likely not the one that cronicle expects and imho this is something that should be handled by cronicle and not the user.

[jhuckaby]

@ioqy I apologize. Node.js recently changed how they do installs. It used to always be installed in /usr/bin/node or /usr/local/bin/node, like for the last 15 years. I never expected this to change, but here we are. This newfangled way of putting it in the home directory under ~/.nvm/versions/node/v{VERSION}/bin/node is new to me, and totally bizarre. I can't figure out why they would do that.

I can try to add some code that "detects" this and throws up a warning at install time.

[ioqy]

No problem. I was just surprised that I ran into such a basic problem by following the installation instructions.

I had a quick look into nvm and control.sh and this should be an easy fix by adding something like the following into control.sh:

[ ! -x "$(which node)" ] && [ -x ~/.nvm/nvm.sh ] && ~/.nvm/nvm.sh use --lts

I'm currently on my phone and didn't check if the command works. Do you want me to try it tomorrow and make a PR?

[jhuckaby]

@ioqy Hmmm, I dunno. I'm not entirely sure what ~/.nvm/nvm.sh use --lts does, and if the user would always want that. They may be running a non-LTS version of Node, or want a specific version to run Cronicle.

nvm is totally new to me, and I'm just starting to look at how it works today.

What we want, I think, is to have nvm simply load itself and make whatever version is currently selected the "live" version, so the correct Node.js binary is added to the path.

It looks like the nvm install process adds this to the ~/.bashrc file for the current user:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

This isn't loaded when systemd boots a startup process, so maybe we just need to call the nvm.sh command without any args? Not sure yet, still researching...

I need to be REALLY careful not to break any existing Cronicle installs.

[jhuckaby]

Okay, so, we're on the right track here, but we need a slightly different command. The nvm.sh script is NOT executable, so we need to test and exec it differently. Also, I think we need to call it without any CLI arguments, so it loads the currently selected "active" version of Node.js. I think something like this may do the trick:

[ ! -x "$(which node)" ] && [ -s ~/.nvm/nvm.sh ] && \. ~/.nvm/nvm.sh

But, the Cronicle control.sh file uses plain shell /bin/sh and not bash, so we may need to modify the command to work under sh.

Confirmed, that command does not work under plain sh, so I need to investigate that. In fact, I think the nvm docs say that it explicitly only works under a few popular shells like bash, zsh and fish w/bass. It probably won't work under /bin/sh at all -- ever.

So I may need to convert Cronicle's 10-year-old control.sh file to use bash instead of sh, and that's going to be a thing.

[jhuckaby]

All fixed in Cronicle v0.9.66. Tested boot on Ubuntu 24 with Node.js LTS installed via nvm.

[ioqy]

Wow, that was quick.

The only thing is that when the nvm default is set to a non LTS version, just sourcing nvm (the command . ~/.nvm/nvm.sh) will then set node to the non LTS version.

You can try this by doing the following:

1. Install the latest non LTS version with nvm install 23
2. Set the default version to 23 with nvm alias default 23
3. Remove nvm from .bashrc
4. Open the shell again and check if node and nvm is not available
5. Run . ~/.nvm/nvm.sh and nvm current which should show version 23

This is why I suggested the parameters use --lts. See https://github.com/nvm-sh/nvm?tab=readme-ov-file#long-term-support

Instead of using . ~/.nvm/nvm.sh you could use . ~/.nvm/nvm.sh && nvm use --lts to use the LTS version.

[jhuckaby]

The only thing is that when the nvm default is set to a non LTS version, just sourcing nvm (the command . ~/.nvm/nvm.sh) will then set node to the non LTS version.

But... this is what we want. Cronicle should use the version of Node.js that is marked as "active" in the nvm system.

Cronicle just executes the same command that nvm puts into the ~/.bashrc, so it simulates what would happen if a user SSH'ed to the server and manually started Cronicle. Systemd doesn't source the bashrc file at boot time, so we're just doing it ourselves instead.

you could use . ~/.nvm/nvm.sh && nvm use --lts to use the LTS version.

Nah. I really don't want Cronicle to switch nvm over to using some other version, outside of the user's control. This seems like a really bad idea to me.

If the user has activated Node.js v23 via nvm, then Cronicle should use Node.js v23 as well. I want the boot behavior to be exactly the same as a user logging in and starting Cronicle manually.

[ioqy]

I just thought that this behavior contradicts the installation instructions which state that only active LTS versions are supported.

[jhuckaby]

@ioqy Ohhhhh! I'm so sorry, I had no idea that verbiage was still in the docs. I'll reword that so it's more of a suggestion rather than a requirement.

It greatly reduces the troubleshooting surface when I know that everyone has Node LTS, but Cronicle should work with the latest.

[jhuckaby]

Reworded. Thanks!