cisagov
Cannot login to elastic dashboard
Cannot login to elastic dashboard. I receieved and error on my dashboard before trying to logout and login again. Upon attempting logon, I receive the message "We couldn't log you in. Please try again.". This was working for around 2 weeks before this error occurred. I did see the error before, but it was resolved with a reboot of the server. I'm sorry, I don't have the error as I can't access the dashboard.
I have verified the password is correct by pulling the passwords again.
I attempted troubleshooting steps in https://cisagov.github.io/lme-docs/docs/markdown/reference/troubleshooting/
To Reproduce
Please complete the following information
Desktop:
- OS: Windows 11
- Browser: Attempted on multiple browsers
Server:
- OS: Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-153-generic x86_64)
athompson@lme:~$ sudo systemctl daemon-reload
sudo systemctl list-unit-files lme\*
[sudo] password for athompson:
UNIT FILE STATE VENDOR PRESET
lme-backups-volume.service generated -
lme-elastalert.service generated -
lme-elasticsearch.service generated -
lme-esdata01-volume.service generated -
lme-fleet-server.service generated -
lme-kibana.service generated -
lme-kibanadata-volume.service generated -
lme-network.service generated -
lme-setup-accts.service generated -
lme-setup-certs.service generated -
lme-wazuh-manager.service generated -
lme.service enabled enabled
12 unit files listed.
athompson@lme:~$ sudo -i podman ps --format "{{.Names}} {{.Status}}"
lme-elasticsearch Up 20 minutes (healthy)
lme-elastalert2 Up 19 minutes
lme-wazuh-manager Up 19 minutes (healthy)
lme-kibana Up 19 minutes (healthy)
lme-fleet-server Up 18 minutes
athompson@lme:~$ curl -k -u elastic:$(sudo -i ansible-vault view /etc/lme/vault/$(sudo -i podman secret ls | grep elastic | awk '{print $1}') | tr -d '\n') https://localhost:9200
{
"name" : "lme-elasticsearch",
"cluster_name" : "LME",
"cluster_uuid" : "QJc9zLnDSV6zrGs-ipAkiQ",
"version" : {
"number" : "8.18.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "04e979aa50b657bebd4a0937389308de82c2bdad",
"build_date" : "2025-04-10T10:09:16.444104780Z",
"build_snapshot" : false,
"lucene_version" : "9.12.1",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
athompson@lme:~$ free -h
df -h
uname -a
lsb_release -a
total used free shared buff/cache available
Mem: 188Gi 35Gi 145Gi 7.0Mi 7.8Gi 151Gi
Swap: 8.0Gi 0B 8.0Gi
Filesystem Size Used Avail Use% Mounted on
tmpfs 19G 2.9M 19G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 98G 85G 8.7G 91% /
tmpfs 95G 84K 95G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sdb2 2.0G 374M 1.5G 21% /boot
tmpfs 19G 4.0K 19G 1% /run/user/1000
Linux lme.cmsec.internal 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
This was because lme_esdata01 filled up went past 90%.
I had to manually delete logs by date: ''' curl -k -u elastic:$(sudo -i ansible-vault view /etc/lme/vault/$(sudo -i podman secret ls | grep elastic | awk '{print $1}')) -XDELETE "https://localhost:9200/wazuh-alerts-4.x-2025.09.16" '''
Reviewing https://cisagov.github.io/lme-docs/docs/markdown/maintenance/index-management/
Update, it happened again today, this is the error. Used the workaround in the article, will attend LME office hours for assistance.
{ "error": { "root_cause": [ { "type": "cluster_block_exception", "reason": "index [.async-search] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block; for more information, see https://www.elastic.co/guide/en/elasticsearch/reference/8.18/fix-watermark-errors.html];" } ], "type": "cluster_block_exception", "reason": "index [.async-search] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block; for more information, see https://www.elastic.co/guide/en/elasticsearch/reference/8.18/fix-watermark-errors.html];" } }
Unable to load visualization Error: [esaggs] > index [.async-search] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block; for more information, see https://www.elastic.co/guide/en/elasticsearch/reference/8.18/fix-watermark-errors.html];
Still receiving the same errors.
Reviewing https://cisagov.github.io/lme-docs/docs/markdown/maintenance/index-management/ Elastic There are quite a few different policies, but the ones that match the instructions the closest have no linked templates.
You may be creating unnecessary shards that are eating up resources and causing you to hit the Watermark error you're seeing. Can you go to Index Management and adjust your index template settings to Replicas: 0? This will avoid duplicating logs moving forward. Linking a related issue that another user had for reference.
