terraform-aws-minikube
terraform-aws-minikube copied to clipboard
After reboot of instance pods are stuck in terminating
Thanks so much for your work on this module, it is in active use at our startup in the creation of ephemeral development environments.
We have our instances set up to restart on a schedule. I notice that after they start back up when we then go to destroy the instance via Terraform the pods are stuck in terminating.
It appears that containerd is running into this issue https://github.com/containerd/containerd/issues/3667 which is specific to Centos 7. The resolution is to toggle on /proc/sys/fs/may_detach_mounts
on startup.
The code I am using to do this is the following:
init-aws-minikube.sh
########################################
########################################
# Set may detach mounts on startup for
# clean termination
########################################
########################################
cat <<EOF | tee /tmp/enable-may-detach-mounts.sh
#!/bin/bash
sudo /bin/sh -c 'echo 1 > /proc/sys/fs/may_detach_mounts'
EOF
chmod +x /tmp/enable-may-detach-mounts.sh
cat <<EOF | tee /etc/systemd/system/detachmounts.service
[Unit]
Description=Set detach mounts to 1
After=network.target
[Service]
Type=simple
ExecStart=/tmp/enable-may-detach-mounts.sh
TimeoutStartSec=0
[Install]
WantedBy=default.target
EOF
systemctl daemon-reload
systemctl enable detachmounts.service
systemctl start detachmounts.service
I'd be happy to contribute a PR to add this if it looks like this is an adequate solution.
Thanks!