Cloudflared service blocking cloud-init service in Ubuntu 20 LTS

[rams3sh]

trafficstars

Describe the bug

I have been trying to bake cloudflared along with applicable configuration as part of hardened OS image. I am relying on AWS services named EC2 image builder which is in turn dependent on a service named cloud-init service within the Ubuntu OS for orchestrating the entire hardening process.

However , I see that build fails every time when I include cloudflared and install it as a service. After 10 days of debugging along with AWS Support folks, finally found that cloudflared service somehow blocks the remaining services (which includes cloud-init) from starting during boot time , resulting in entire pipeline getting timed out and failing.

Apparently it seems , systemd waits on cloudflared service to complete before executing other dependent services , but cloudflared would keep running (which is expected) , while other services are in waiting state (as given in below code block).

I have included a small snippet of terminal output while cloudflared service was running (this is after two hours of the pipeline initialisation).

At first, cloud-init status is checked if the initialization was completed. It keeps waiting. The command is interrupted. Then the cloudflared service is cancelled and the cloud-init status is checked again and it outputs as done.

temp@ip-172-31-32-190:~$ cloud-init status --wait
......................................................................................................................................................................^CTraceback (most recent call last):
  File "/usr/bin/cloud-init", line 11, in <module>
    load_entry_point('cloud-init==20.3', 'console_scripts', 'cloud-init')()
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 891, in main
    retval = util.log_time(
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2292, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 61, in handle_status_args
    sleep(0.25)
  File "/usr/lib/python3/dist-packages/cloudinit/signal_handler.py", line 48, in _handle_exit
    util.multi_log(contents.getvalue(),
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 400, in multi_log
    with open(conpath, 'w') as wfh:
PermissionError: [Errno 13] Permission denied: '/dev/console'

temp@ip-172-31-32-190:~$ sudo systemctl list-jobs
[sudo] password for temp: 
JOB UNIT                                 TYPE  STATE  
157 cloud-final.service                  start waiting
158 systemd-update-utmp-runlevel.service start waiting
2   multi-user.target                    start waiting
1   graphical.target                     start waiting
109 cloudflared.service                  start running
151 cloud-init.target                    start waiting

6 jobs listed.
temp@ip-172-31-32-190:~$ sudo systemctl cancel 109
temp@ip-172-31-32-190:~$ cloud-init status --wait

status: done

My systemctl configuration looks like this :-

[Unit]
Description=cloudflared
After=network.target

[Service]
TimeoutStartSec=0
Type=notify
ExecStart=/usr/bin/cloudflared --no-autoupdate --config /etc/cloudflared/config.yml tunnel run
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Expected behavior

cloudflared service should run as daemon without blocking other services from getting executed.

Environment and versions

• OS: Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1028-aws x86_64)
• Architecture: [e.g. AMD, ARM] AMD64
• Version: [e.g. 2022.02.0] 2022.5.3 (used in most recent attempt of the troubleshooting exercise)

[sudarshan-reddy]

Hey @rams3sh ,

Starting a service can be tricky because when executing cloud-init configuration/scripts you are essentially already within a systemd unit while you try to start another systemd unit.

Have you tried systemctl using

--no-block
           Do not synchronously wait for the requested operation to
           finish. If this is not specified, the job will be verified,
           enqueued and systemctl will wait until the unit's start-up is
           completed. By passing this argument, it is only verified and
           enqueued. This option may not be combined with --wait.

[rams3sh]

Hey @sudarshan-reddy

I did'nt get the last solution. Did you mean to say that I should be adding --no-block to the services startup command that are waiting for cloudflared service to complete . If yes, I use vanilla AWS provided ubuntu image for this. I am little skeptical on controlling other services and their startup configurations as there may be dependency on AWS services and it may result in other weird issues if configured in a non-native way. There is'nt much documentation on the AWS dependencies either for troubleshooting if any issue were to occur on that end.

I currently depend on cloudflared service install command to setup all startup configurations of cloudflared. If there exists any params that can be added / modified / removed from / to any of these configurations to make systemd not wait for cloudflared to exit, it would be great. I would be more comfortable in modifying the cloudflared related configs than other service configs to solve this issue.

[rams3sh]

@abelinkinbio / @sudarshan-reddy

Would you guys know any solution to this ? .. I was unable to determine any solution for the issue.

[sudarshan-reddy]

@rams3sh : the solution is to use --no-block in your cloud-init script.