influxdb
influxdb copied to clipboard
influxdata
startup script influxd-systemd-start.sh stuck in while loop on slower servers
This is a continuation of issue #21967 which has now been partially resolved. This comment instructed users who are still experiencing errors in 1.8.10 to open a new ticket.
Steps to reproduce: List the minimal actions needed to reproduce the behavior.
- Get a slow server, such as a raspberry pi
- Attempt to start influxdb (
systemctl start influxdb) - Observe the failure:
Job for influxdb.service failed because a timeout was exceeded.
Expected behavior: Service starts in less time than the timeout
Actual behavior: Service was killed when it was partially started. Because the service is set to restart on failure, it will then repeated attempt to start and fail each time.
Environment info:
- System info: Linux 5.4.0-88-generic x86_64
- InfluxDB version: InfluxDB v1.8.10 (git: 1.8 688e697c51fd)
- Other relevant environment details: Installed on a VM on an older server. They process does start and get the health check endpoint up and running, it just takes a long time
Additional info: The solution is to increase the timeout in influxdb.service, which was suggested here, confirmed to be working here, here and I have also confirmed that this completely solves the issue.
The value of 120s (or 2m) works on my system sometimes. Others have reported this value has word for them. There is really no penalty for setting it higher, as the only time it will ever hit this limit would be that the server is slow or there is some error where the service does not exit, but is also never going to come online. I'd suggest making the timeout 3m.
Keep in mind this is not a "slow" server problem but size of data base.
Once you reach huge DB size it takes long time to load all shards no matter how fast your server is.
For example, on our setup with >4TB of data it takes about 10 min before influxd loads all shards and then starts http service so that health check works.
This means I'd have to set systemd start timeout TimeoutStartSec to something like 900.
But what if my data grows 2x then even 900s will not be enough.
It looks its best to set TimeoutStartSec to "infinity" to disable the timeout logic altogether
I'm also facing this issue on a raspberry pi 2. I managed to install influx without an issue, but after some data accumulated in the dbs, influx fails to start after a system reboot fe.
$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs
$ uname -a
Linux raspberrypi 5.10.63+ #1496 Wed Dec 1 15:57:05 GMT 2021 armv6l GNU/Linux
$ dpkg -l | grep influx
ii influxdb 1.8.10-1 armhf Distributed time-series database.
I solved it by altering:
cat /etc/systemd/system/multi-user.target.wants/influxdb.service
# If you modify this, please also make sure to edit init.sh
[Unit]
Description=InfluxDB is an open-source, distributed, time series database
Documentation=https://docs.influxdata.com/influxdb/
After=network-online.target
[Service]
User=influxdb
Group=influxdb
LimitNOFILE=65536
EnvironmentFile=-/etc/default/influxdb
ExecStart=/usr/lib/influxdb/scripts/influxd-systemd-start.sh
KillMode=control-group
Restart=on-failure
#Type=forking
PIDFile=/var/lib/influxdb/influxd.pid
TimeoutStartSec=120
[Install]
WantedBy=multi-user.target
Alias=influxd.service
(Commented Type=forking, introduced TimeoutStartSec=120) Credits of this fix: @jriobello - https://github.com/influxdata/influxdb/issues/21967#issuecomment-945719018
I just wanted to add that I came across this page after a search for "InfluxDB API unavailable after". My setup is also largely untouched and Influxdb was working prior to reboot, but after a reboot it was stuck in the activating state. I am pretty good about keeping it up to date and was already running the 1.8.10 version so although I do recall seeing influxdb restarting in previous versions, for my setup it seems that InfluxDB has just started taking longer to start. At this point it could be due to the size of the DB. Mine is ~200 Gig's, but I'm not sure.
Anyhow I seeing the influxd-systemd-start.sh script continuously looping and I noticed it would loop at the 86th attempt I was able to resolve it by adding the longer startup timeout ( I set mine to 120 seconds) as described in issue 22803 here: https://github.com/influxdata/influxdb/issues/22803
Before [judsonm@Cent7Test]:/home/judsonm>sudo yum list influxdb Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile
- base: mirror.pit.teraswitch.com
- epel: repos.eggycrew.com
- extras: nc-centos-mirror.iwebfusion.net
- updates: mirror.ilumno.com Installed Packages influxdb.x86_64 1.8.10-1 @influxdb [judsonm@Cent7Test]:/home/judsonm>
Feb 7 10:46:17 Cent7Test influxd-systemd-start.sh[20506]: InfluxDB API unavailable after 85 attempts... Feb 7 10:46:18 Cent7Test influxd-systemd-start.sh[20506]: ts=2022-02-07T15:46:18.156772Z lvl=info msg="Opened shard" log_id=0ZXDaBrl000 service=store trace_id=0ZXDaCJ0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/telegraf/autogen/1222 duration=3214.365ms Feb 7 10:46:18 Cent7Test influxd-systemd-start.sh[20506]: ts=2022-02-07T15:46:18.417608Z lvl=info msg="Opened file" log_id=0ZXDaBrl000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/telegraf/autogen/1290/000009196-000000002.tsm id=0 duration=259.313ms Feb 7 10:46:18 Cent7Test influxd-systemd-start.sh[20506]: InfluxDB API unavailable after 86 attempts... Feb 7 10:46:18 Cent7Test systemd[1]: influxdb.service start operation timed out. Terminating. Feb 7 10:46:19 Cent7Test systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database. Feb 7 10:46:19 Cent7Test systemd[1]: Unit influxdb.service entered failed state. Feb 7 10:46:19 Cent7Test systemd[1]: influxdb.service failed. Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: Merging with configuration at: /etc/influxdb/influxdb.conf Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.592193Z lvl=info msg="InfluxDB starting" log_id=0ZXDfkI0000 version=1.8.10 branch=1.8 commit=688e697c51fd Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.592221Z lvl=info msg="Go runtime" log_id=0ZXDfkI0000 version=go1.13.8 maxprocs=4 Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: Merging with configuration at: /etc/influxdb/influxdb.conf Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: InfluxDB API unavailable after 1 attempts... Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.698705Z lvl=info msg="Using data dir" log_id=0ZXDfkI0000 service=store path=/var/lib/influxdb/data Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.698742Z lvl=info msg="Compaction settings" log_id=0ZXDfkI0000 service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648 Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.698754Z lvl=info msg="Open store (start)" log_id=0ZXDfkI0000 service=store trace_id=0ZXDfkhW000 op_name=tsdb_open op_event=start Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.775833Z lvl=info msg="Opened file" log_id=0ZXDfkI0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/1351/000000003-000000002.tsm id=0 duration=21.120ms Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.775972Z lvl=info msg="Opened file" log_id=0ZXDfkI0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/1341/000000062-000000002.tsm id=0 duration=25.299ms Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.775996Z lvl=info msg="Reading file" log_id=0ZXDfkI0000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1351/_00016.wal size=10583641 Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.786360Z lvl=info msg="Opened file" log_id=0ZXDfkI0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/1339/000000062-000000002.tsm id=0 duration=34.111ms Feb 7 10:46:19 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:19.795010Z lvl=info msg="Opened file" log_id=0ZXDfkI0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/1343/000000062-000000002.tsm id=0 duration=42.740ms Feb 7 10:46:20 Cent7Test influxd-systemd-start.sh[21264]: ts=2022-02-07T15:46:20.327701Z lvl=info msg="Reading file" log_id=0ZXDfkI0000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1351/_00017.wal size=10586036 Feb 7 10:46:20 Cent7Test influxd-systemd-start.sh[21264]: InfluxDB API unavailable after 2 attempts...
edited the influxd.service to add the TimeoutStartSec=120......
[judsonm@Cent7Test]:/home/judsonm>cat /etc/systemd/system/influxd.service
[Unit] Description=InfluxDB is an open-source, distributed, time series database Documentation=https://docs.influxdata.com/influxdb/ After=network-online.target
[Service] User=influxdb Group=influxdb LimitNOFILE=65536 EnvironmentFile=-/etc/default/influxdb ExecStart=/usr/lib/influxdb/scripts/influxd-systemd-start.sh KillMode=control-group Restart=on-failure Type=forking PIDFile=/var/lib/influxdb/influxd.pid TimeoutStartSec=120
[Install] WantedBy=multi-user.target Alias=influxd.service [judsonm@Cent7Test]:/home/judsonm>
Finally able to start after the 110th attempt:
Feb 7 11:24:04 Cent7Test influxd-systemd-start.sh[5254]: InfluxDB API unavailable after 109 attempts... Feb 7 11:24:04 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:04.749828Z lvl=info msg="Opened shard" log_id=0ZXFj3l0000 service=store trace_id=0ZXFj49G000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/telegraf/autogen/1320 duration=3422.958ms Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.554035Z lvl=info msg="Opened shard" log_id=0ZXFj3l0000 service=store trace_id=0ZXFj49G000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/telegraf/autogen/1262 duration=2319.729ms Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: InfluxDB API unavailable after 110 attempts... Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.898781Z lvl=info msg="Open store (end)" log_id=0ZXFj3l0000 service=store trace_id=0ZXFj49G000 op_name=tsdb_open op_event=end op_elapsed=114661.413ms Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.901623Z lvl=info msg="Opened service" log_id=0ZXFj3l0000 service=subscriber Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.901652Z lvl=info msg="Starting monitor service" log_id=0ZXFj3l0000 service=monitor Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.901664Z lvl=info msg="Registered diagnostics client" log_id=0ZXFj3l0000 service=monitor name=build Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.901672Z lvl=info msg="Registered diagnostics client" log_id=0ZXFj3l0000 service=monitor name=runtime Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.901676Z lvl=info msg="Registered diagnostics client" log_id=0ZXFj3l0000 service=monitor name=network Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.901699Z lvl=info msg="Registered diagnostics client" log_id=0ZXFj3l0000 service=monitor name=system Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.901727Z lvl=info msg="Starting precreation service" log_id=0ZXFj3l0000 service=shard-precreation check_interval=10m advance_period=30m Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.902650Z lvl=info msg="Storing statistics" log_id=0ZXFj3l0000 service=monitor db_instance=_internal db_rp=monitor interval=10s Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.903930Z lvl=info msg="Starting snapshot service" log_id=0ZXFj3l0000 service=snapshot Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.903951Z lvl=info msg="Starting continuous query service" log_id=0ZXFj3l0000 service=continuous_querier Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.903968Z lvl=info msg="Starting HTTP service" log_id=0ZXFj3l0000 service=httpd authentication=false Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.903975Z lvl=info msg="opened HTTP access log" log_id=0ZXFj3l0000 service=httpd path=stderr Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.911032Z lvl=info msg="Listening on HTTP" log_id=0ZXFj3l0000 service=httpd addr=0.0.0.0:8086 https=false Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.911067Z lvl=info msg="Starting retention policy enforcement service" log_id=0ZXFj3l0000 service=retention check_interval=30m Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.911570Z lvl=info msg="Listening for signals" log_id=0ZXFj3l0000 Feb 7 11:24:05 Cent7Test influxd-systemd-start.sh[5254]: ts=2022-02-07T16:24:05.918103Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0ZXFj3l0000 Feb 7 11:24:06 Cent7Test influxd-systemd-start.sh[5254]: [httpd] 127.0.0.1 - - [07/Feb/2022:11:24:06 -0500] "GET /health HTTP/1.1" 200 107 "-" "curl/7.29.0" 5f24f92d-8832-11ec-8001-0050569bd466 241 Feb 7 11:24:06 Cent7Test influxd-systemd-start.sh[5254]: InfluxDB started
This problem can also lead to influxdb.service not starting at all on server boot (inactive (dead)):
+phd-mon:~# systemctl --all --failed
0 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
-phd-mon:~# systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/influxdb.service.d
└─override.conf
Active: inactive (dead)
Docs: https://docs.influxdata.com/influxdb/
-phd-mon:~# systemctl restart influxdb.service
Job for influxdb.service failed because a timeout was exceeded.
See "systemctl status influxdb.service" and "journalctl -xe" for details.
-phd-mon:~# journalctl -xe
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.033416Z lvl=info msg="Open store (end)" log_id=0Zk1JU6l000
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.070901Z lvl=info msg="Opened service" log_id=0Zk1JU6l000 s
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.070933Z lvl=info msg="Starting monitor service" log_id=0Zk
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.070943Z lvl=info msg="Registered diagnostics client" log_i
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.070949Z lvl=info msg="Registered diagnostics client" log_i
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.070962Z lvl=info msg="Registered diagnostics client" log_i
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.070967Z lvl=info msg="Registered diagnostics client" log_i
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.070995Z lvl=info msg="Starting precreation service" log_id
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.071009Z lvl=info msg="Starting snapshot service" log_id=0Z
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.071020Z lvl=info msg="Starting continuous query service" l
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.071054Z lvl=info msg="Storing statistics" log_id=0Zk1JU6l0
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.095516Z lvl=info msg="Starting HTTP service" log_id=0Zk1JU
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.095756Z lvl=info msg="Listening on HTTP" log_id=0Zk1JU6l00
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.116248Z lvl=info msg="Starting retention policy enforcemen
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.116286Z lvl=info msg="Starting OpenTSDB service" log_id=0Z
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.116433Z lvl=info msg="Listening on TCP" log_id=0Zk1JU6l000
Feb 18 10:09:52 phd-mon influxd[5363]: ts=2022-02-18T09:09:52.116685Z lvl=info msg="Listening for signals" log_id=0Zk1JU
Feb 18 10:09:52 phd-mon influxd[5363]: InfluxDB started
Feb 18 10:09:52 phd-mon systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
-- Subject: A start job for unit influxdb.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit influxdb.service has finished successfully.
--
-- The job identifier is 1299.
Feb 18 10:10:00 phd-mon influxd[5363]: ts=2022-02-18T09:10:00.139303Z lvl=info msg="Continuous query execution (start)"
Feb 18 10:10:00 phd-mon influxd[5363]: ts=2022-02-18T09:10:00.139358Z lvl=info msg="Executing continuous query" log_id=0
Feb 18 10:10:01 phd-mon CRON[5652]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb 18 10:10:01 phd-mon CRON[5653]: (root) CMD (if [ -x /usr/bin/mrtg ] && [ -r /etc/mrtg.cfg ] && [ -d "$(grep '^[[:spa
Feb 18 10:10:06 phd-mon influxd[5363]: ts=2022-02-18T09:10:06.846256Z lvl=info msg="Finished continuous query" log_id=0Z
Feb 18 10:10:06 phd-mon influxd[5363]: ts=2022-02-18T09:10:06.846362Z lvl=info msg="Continuous query execution (end)" lo
Feb 18 10:10:06 phd-mon influxd[5363]: ts=2022-02-18T09:10:06.846596Z lvl=info msg="Continuous query execution (start)"
Feb 18 10:10:06 phd-mon influxd[5363]: ts=2022-02-18T09:10:06.846613Z lvl=info msg="Executing continuous query" log_id=0
Feb 18 10:10:10 phd-mon CRON[5652]: pam_unix(cron:session): session closed for user root
Feb 18 10:10:13 phd-mon influxd[5363]: ts=2022-02-18T09:10:13.354187Z lvl=info msg="Finished continuous query" log_id=0Z
Feb 18 10:10:13 phd-mon influxd[5363]: ts=2022-02-18T09:10:13.354252Z lvl=info msg="Continuous query execution (end)" lo
Feb 18 10:10:13 phd-mon influxd[5363]: ts=2022-02-18T09:10:13.354416Z lvl=info msg="Continuous query execution (start)"
Feb 18 10:10:13 phd-mon influxd[5363]: ts=2022-02-18T09:10:13.354447Z lvl=info msg="Executing continuous query" log_id=0
Feb 18 10:10:15 phd-mon systemd[1]: [email protected]: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit [email protected] has successfully entered the 'dead' state.
Feb 18 10:10:15 phd-mon systemd[1]: [email protected]: Service RestartSec=100ms expired, scheduling restart.
Feb 18 10:10:15 phd-mon systemd[1]: [email protected]: Scheduled restart job, restart counter is at 3.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Automatic restarting of the unit [email protected] has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Feb 18 10:10:15 phd-mon systemd[1]: Stopped Serial Getty on ttyS0.
-- Subject: A stop job for unit [email protected] has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A stop job for unit [email protected] has finished.
--
-- The job identifier is 1416 and the job result is done.
Feb 18 10:10:15 phd-mon systemd[1]: Started Serial Getty on ttyS0.
-- Subject: A start job for unit [email protected] has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit [email protected] has finished successfully.
--
-- The job identifier is 1416.
+phd-mon:~# systemctl status influxdb.service
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/influxdb.service.d
└─override.conf
Active: active (running) since Fri 2022-02-18 10:09:52 CET; 45s ago
Docs: https://docs.influxdata.com/influxdb/
Process: 5363 ExecStart=/usr/lib/influxdb/scripts/influxd-systemd-start.sh (code=exited, status=0/SUCCESS)
Main PID: 5364 (influxd)
Tasks: 23 (limit: 19660)
Memory: 12.3G
CGroup: /system.slice/influxdb.service
└─5364 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
Feb 18 10:10:13 phd-mon influxd[5363]: ts=2022-02-18T09:10:13.354252Z lvl=info msg="Continuous query execution (end)" lo
Feb 18 10:10:13 phd-mon influxd[5363]: ts=2022-02-18T09:10:13.354416Z lvl=info msg="Continuous query execution (start)"
Feb 18 10:10:13 phd-mon influxd[5363]: ts=2022-02-18T09:10:13.354447Z lvl=info msg="Executing continuous query" log_id=0
Feb 18 10:10:19 phd-mon influxd[5363]: [tcp] 2022/02/18 10:10:19 tcp.Mux: handler not registered: 71. Connection from 12
Feb 18 10:10:25 phd-mon influxd[5363]: ts=2022-02-18T09:10:25.803649Z lvl=info msg="Finished continuous query" log_id=0Z
Feb 18 10:10:25 phd-mon influxd[5363]: ts=2022-02-18T09:10:25.803716Z lvl=info msg="Continuous query execution (end)" lo
Feb 18 10:10:25 phd-mon influxd[5363]: ts=2022-02-18T09:10:25.803887Z lvl=info msg="Continuous query execution (start)"
Feb 18 10:10:25 phd-mon influxd[5363]: ts=2022-02-18T09:10:25.803900Z lvl=info msg="Executing continuous query" log_id=0
Feb 18 10:10:37 phd-mon influxd[5363]: ts=2022-02-18T09:10:37.054791Z lvl=info msg="Finished continuous query" log_id=0Z
Feb 18 10:10:37 phd-mon influxd[5363]: ts=2022-02-18T09:10:37.055764Z lvl=info msg="Continuous query execution (end)" lo
Version: 1.8.10-1
Any chance of getting the merge request reviewed so we can close this ticket?
Hi :)
I have added the TimeoutStartSec to my influxdb.service and now when the service is started it reports 'Unknown Key name 'TimeoutStartSec' in section 'Service', Ignore...'
The service started though but just curious if anyone saw this when checking the status?
It sounds like you are using an old version of systemd that is ignoring the timeout on the start time.
As you noted, it won't hurt anything, but it likely won't solve the problem of the service not starting faster than systemd is willing to wait (assuming you were having that issue).
If the database grows to be too big to load within the default timeout, you'll need to upgrade unless someone can find a workaround. Upgrading is frequently a good idea, but sometimes that isn't an option for one reason or another.
The latest stable version of systemd in Debian is v247 (at time of writing). You can check to see what version you have with systemctl --version
I expect that you'll have something older than that.
On September 15, 2022 7:06:44 PM CDT, Dave Smith @.***> wrote:
Hi :)
I have added the TimeoutStartSec to my influxdb.service and now when the service is started it reports 'Unknown Key name 'TimeoutStartSec' in section 'Service', Ignore...'
The service started though but just curious if anyone saw this when checking the status?
-- Reply to this email directly or view it on GitHub: https://github.com/influxdata/influxdb/issues/22803#issuecomment-1248770869 You are receiving this because you authored the thread.
Message ID: @.***> -- Sent from my iPod. Please excuse my brevity.
OK thanks very much. I will check on the version and update as needed.
Cheers!
From: anon8675309 @.> Sent: Thursday, 22 September 2022 12:23 PM To: influxdata/influxdb @.> Cc: Dave Smith @.>; Comment @.> Subject: Re: [influxdata/influxdb] startup script influxd-systemd-start.sh stuck in while loop on slower servers (Issue #22803)
It sounds like you are using an old version of systemd that is ignoring the timeout on the start time.
As you noted, it won't hurt anything, but it likely won't solve the problem of the service not starting faster than systemd is willing to wait (assuming you were having that issue).
If the database grows to be too big to load within the default timeout, you'll need to upgrade unless someone can find a workaround. Upgrading is frequently a good idea, but sometimes that isn't an option for one reason or another.
The latest stable version of systemd in Debian is v247 (at time of writing). You can check to see what version you have with systemctl --version
I expect that you'll have something older than that.
On September 15, 2022 7:06:44 PM CDT, Dave Smith @.***mailto:***@***.***> wrote:
Hi :)
I have added the TimeoutStartSec to my influxdb.service and now when the service is started it reports 'Unknown Key name 'TimeoutStartSec' in section 'Service', Ignore...'
The service started though but just curious if anyone saw this when checking the status?
-- Reply to this email directly or view it on GitHub: https://github.com/influxdata/influxdb/issues/22803#issuecomment-1248770869 You are receiving this because you authored the thread.
Message ID: @.***mailto:***@***.***> -- Sent from my iPod. Please excuse my brevity.
Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Finfluxdata%2Finfluxdb%2Fissues%2F22803%23issuecomment-1254431592&data=05%7C01%7C%7C0d493351e0a6450e637008da9c415c38%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637994101783488814%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=15pjUmTINRynJh%2FgiXDJ6PFOOU38w93bazJcLizF2tE%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA2R3NA6T7TBMBMFFVYLWCLV7O7H7ANCNFSM5HC5XG7Q&data=05%7C01%7C%7C0d493351e0a6450e637008da9c415c38%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637994101783488814%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=i8bioLQdb0J1uJG6v3suG63XKO9H0IXi8JNMkHcCDTc%3D&reserved=0. You are receiving this because you commented.Message ID: @.@.>>
This will be fixed if https://github.com/influxdata/influxdb/pull/23400 gets merged into a release