logdna-agent
logdna-agent copied to clipboard
logdna
Agent not starting on Elastic Beanstalk instance
I'm running an app on AWS Elastic Beanstalk running Amazon Linux 2 with Docker platform. I have changed deployment to immutable, so that with every app update a new EC2 instance is spawned as I currently have only one instance running and do want minimal downtime. Unfortunately the LogDNA Agent does not start up and after every deployment I have to start it manually (sudo systemctl start logdna-agent).
I had used the script v1 and also tried to use services key or platform hooks but nothing worked.
Is there anything I can do or debug why the service is not starting?
@TJDev i recommend to check logdna-agent service log:
journalctl -u logdna-agent
I assume the service is enabled:
systemctl list-unit-files | grep logdna-agent
May be start condition is not working. Unit file says to start after network.target:
$ cat logdna-agent.service
[Unit]
Description=Logdna Agent
Documentation=https://docs.logdna.com
After=network.target
is it started?
systemctl | grep network.target
to see whole journal log since last boot:
journalctl -b
I assume the service is enabled:
systemctl list-unit-files | grep logdna-agent
There is no entry for logdna-agent.
is it started?
systemctl | grep network.target
Network is started
What happens if you try to enable the service and test it again?
systemctl enable logdna-agent
sh-4.2$ sudo systemctl list-unit-files | grep logdna-agent
sh-4.2$ sudo systemctl enable logdna-agent
logdna-agent.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig logdna-agent on
sh-4.2$ sudo systemctl list-unit-files | grep logdna-agent
sh-4.2$ sudo systemctl status logdna-agent
● logdna-agent.service - LSB: Starts LogDNA Agent at boot
Loaded: loaded (/etc/rc.d/init.d/logdna-agent; bad; vendor preset: disabled)
Active: inactive (dead)
Docs: man:systemd-sysv-generator(8)
sh-4.2$ sudo systemctl start logdna-agent
sh-4.2$ sudo systemctl status logdna-agent
● logdna-agent.service - LSB: Starts LogDNA Agent at boot
Loaded: loaded (/etc/rc.d/init.d/logdna-agent; bad; vendor preset: disabled)
Active: active (exited) since Mon 2022-12-05 09:36:07 UTC; 2s ago
Docs: man:systemd-sysv-generator(8)
Process: 14144 ExecStart=/etc/rc.d/init.d/logdna-agent start (code=exited, status=0/SUCCESS)
Dec 05 09:36:07 systemd[1]: Starting LSB: Starts LogDNA Agent at boot...
Dec 05 09:36:07 logdna-agent[14144]: Already started
Dec 05 09:36:07 systemd[1]: Started LSB: Starts LogDNA Agent at boot.
my bad, "old" agent, Sys-V based service. the log goes to:
/var/log/logdna-agent.log
@dkhokhlov Sorry for the late response. Unfortunately the issue is still open. The agent does not start after deployment.
2023-03-30 12:24:50 0000 [info] logdna-agent/2.2.1 started on ELB-env (x.x.x.x) 2023-03-30 12:24:50 0000 [info] LogDNA URL: https://logs.logdna.com:443/logs/agent 2023-03-30 12:24:50 0000 [info] streaming /var/log: 25 file(s) 2023-03-30 12:25:50 0000 [info] streaming /var/log: 6 new file(s), 31 total file(s) 2023-03-30 12:25:54 0000 [info] got SIGTERM signal, shutting down... 2023-03-30 12:42:57 0000 [info] logdna-agent/2.2.1 started on ELB-env (x.x.x.x) 2023-03-30 12:42:57 0000 [info] LogDNA URL: https://logs.logdna.com:443/logs/agent 2023-03-30 12:42:57 0000 [info] streaming /var/log: 32 file(s)
Deployment on 12:23, manual restart on 12:42
there was agent shutdown - SIGTERM, graceful stop request before you started it again. that is the reason agent was not running when you were looking. I would check system logs. runlevel changes. also check runlevel of agent service.
I expect the SIGTERM has a relation to the shutdown of cfn-hub as they happen at the same time.
sh-4.2$ sudo cat /var/log/messages | grep 08:13
Apr 21 08:13:36 ip-172-31-37-214 systemd: Stopping This is cfn-hup daemon...
Apr 21 08:13:43 ip-172-31-37-214 dhclient[3120]: XMT: Solicit on eth0, interval 111720ms.
Apr 21 08:13:46 ip-172-31-37-214 systemd: Stopped This is cfn-hup daemon.
2023-04-21 08:13:36 0000 [info] got SIGTERM signal, shutting down...
cfn-hub is about AWS config changes...
lets temporary try to disable agent stop in init script to isolate - comment this whole section out in /etc/init.d/logdna-agent:
https://github.com/logdna/logdna-agent/blob/f0d219d91c08ac0f7c9299a99e2b9ef9ac6d707d/tools/files/linux/init-script#L76
expected:
- agent will keep running
lets keep top if and echo and remove this: https://github.com/logdna/logdna-agent/blob/f0d219d91c08ac0f7c9299a99e2b9ef9ac6d707d/tools/files/linux/init-script#L78-L102
I will switch to logdna-agent-v2 as I have also out of memory exceptions for time to time and the systemd unit file should restart the agent in that case.