Investigate failure to login to Grafana prod
Investigate "database or disk is full" error when logging to Grafana prod
wiki page for investigating Grafana issues: https://dnceng.visualstudio.com/internal/_wiki/wikis/DNCEng%20Services%20Wiki/977/-Alerts-Azure-Monitor-Alert-Grafana-health-(Availability-Alert)
The memory metric doesn't indicate any problems
Previous similar issue: https://github.com/dotnet/dnceng/issues/959
[email protected]@grafana8-prod:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 29G 0 100% /
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 1.6G 150M 1.5G 10% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/loop0 64M 64M 0 100% /snap/core20/2264
/dev/loop1 64M 64M 0 100% /snap/core20/2318
/dev/loop3 92M 92M 0 100% /snap/lxd/24061
/dev/loop4 39M 39M 0 100% /snap/snapd/21759
/dev/loop5 39M 39M 0 100% /snap/snapd/21465
/dev/sda15 105M 6.1M 99M 6% /boot/efi
/dev/sdb1 32G 17G 14G 54% /mnt
/dev/loop6 92M 92M 0 100% /snap/lxd/29619
tmpfs 1.6G 0 1.6G 0% /run/user/15740529
We doubled the disk space from the azure portal and were able to logon to the site after the disk size was increased:
Filesystem Size Used Avail Use% Mounted on /dev/root 62G 29G 33G 47% / devtmpfs 7.9G 0 7.9G 0% /dev tmpfs 7.9G 0 7.9G 0% /dev/shm tmpfs 1.6G 1.2M 1.6G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup /dev/loop0 64M 64M 0 100% /snap/core20/2318 /dev/loop3 92M 92M 0 100% /snap/lxd/24061 /dev/loop1 64M 64M 0 100% /snap/core20/2264 /dev/loop2 92M 92M 0 100% /snap/lxd/29619 /dev/loop4 39M 39M 0 100% /snap/snapd/21465 /dev/loop5 39M 39M 0 100% /snap/snapd/21759 /dev/sda15 105M 6.1M 99M 6% /boot/efi /dev/sdb1 32G 17G 14G 54% /mnt tmpfs 1.6G 0 1.6G 0% /run/user/13496015
Closing alert