dnceng icon indicating copy to clipboard operation
dnceng copied to clipboard

Investigate failure to login to Grafana prod

Open MilenaHristova opened this issue 1 year ago • 2 comments

Investigate "database or disk is full" error when logging to Grafana prod image

wiki page for investigating Grafana issues: https://dnceng.visualstudio.com/internal/_wiki/wikis/DNCEng%20Services%20Wiki/977/-Alerts-Azure-Monitor-Alert-Grafana-health-(Availability-Alert)

The memory metric doesn't indicate any problems

MilenaHristova avatar Aug 19 '24 15:08 MilenaHristova

Previous similar issue: https://github.com/dotnet/dnceng/issues/959

MilenaHristova avatar Aug 19 '24 15:08 MilenaHristova

[email protected]@grafana8-prod:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G   29G     0 100% /
devtmpfs        7.9G     0  7.9G   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           1.6G  150M  1.5G  10% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/loop0       64M   64M     0 100% /snap/core20/2264
/dev/loop1       64M   64M     0 100% /snap/core20/2318
/dev/loop3       92M   92M     0 100% /snap/lxd/24061
/dev/loop4       39M   39M     0 100% /snap/snapd/21759
/dev/loop5       39M   39M     0 100% /snap/snapd/21465
/dev/sda15      105M  6.1M   99M   6% /boot/efi
/dev/sdb1        32G   17G   14G  54% /mnt
/dev/loop6       92M   92M     0 100% /snap/lxd/29619
tmpfs           1.6G     0  1.6G   0% /run/user/15740529

MilenaHristova avatar Aug 19 '24 16:08 MilenaHristova

We doubled the disk space from the azure portal and were able to logon to the site after the disk size was increased:

Filesystem Size Used Avail Use% Mounted on /dev/root 62G 29G 33G 47% / devtmpfs 7.9G 0 7.9G 0% /dev tmpfs 7.9G 0 7.9G 0% /dev/shm tmpfs 1.6G 1.2M 1.6G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup /dev/loop0 64M 64M 0 100% /snap/core20/2318 /dev/loop3 92M 92M 0 100% /snap/lxd/24061 /dev/loop1 64M 64M 0 100% /snap/core20/2264 /dev/loop2 92M 92M 0 100% /snap/lxd/29619 /dev/loop4 39M 39M 0 100% /snap/snapd/21465 /dev/loop5 39M 39M 0 100% /snap/snapd/21759 /dev/sda15 105M 6.1M 99M 6% /boot/efi /dev/sdb1 32G 17G 14G 54% /mnt tmpfs 1.6G 0 1.6G 0% /run/user/13496015

Closing alert

davfost avatar Aug 20 '24 00:08 davfost