Index / Visualization Mismatch

[koelslaw]

Indexes are present but the dashboards error out.

health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_7.12.1_001                ordWlk6IRyK7laqYYdmhhQ   1   0        930          996      7.5mb          7.5mb
green  open   .monitoring-kibana-7-2021.05.16   3W3wuq_LQJGcf_OllHxGdg   1   0        348            0    189.2kb        189.2kb
green  open   ecs-suricata-network-2021.05.16   gUPiWrOlRoOzADHtwsqh6Q   1   0        489            0      2.4mb          2.4mb
green  open   ecs-zeek-observations-2021.05.16  bNo206X4TD2ycj3Dfyrnzw   1   0          2            0     18.3kb         18.3kb
green  open   .apm-agent-configuration          8eMb3Pl4QoaWOJ_beMO6bA   1   0          0            0       208b           208b
green  open   .kibana_task_manager_7.12.1_001   9LIMNTACSdqCLStf82nM_Q   1   0          9          199    280.7kb        280.7kb
green  open   ecs-zeek-miscellaneous-2021.05.16 gZeD5cGRRSy-voYkG08vpA   1   0         23            0     56.6kb         56.6kb
green  open   ecs-zeek-network-2021.05.16       r_TV1DqkT1-CeounuVTymQ   1   0         16            0     71.3kb         71.3kb
green  open   .kibana-event-log-7.12.1-000001   K0ny7VC5TkeMfSKEXukYOg   1   0          2            0       11kb           11kb
green  open   .tasks                            roxuTzyEQzedwrMMjKxiwA   1   0          2            0     13.7kb         13.7kb
green  open   .monitoring-logstash-7-2021.05.16 xEgfFpBOQcGxpOLJShZxmw   1   0      74406            0     29.5mb         29.5mb
green  open   ecs-zeek-diagnostic-2021.05.16    M9kC17OKR3Oe-oFFZCnacQ   1   0       1762            0      319kb          319kb
green  open   .apm-custom-link                  A0O5HyMPTXmvuGoL3_iz3Q   1   0          0            0       208b           208b
green  open   .async-search                     GYZJhfZNQkW_YFS6khIDmg   1   0         15            0     12.7kb         12.7kb
green  open   .monitoring-es-7-2021.05.16       bN0BQnOBTQSNwPvq0pxAXg   1   0       4385          488      5.2mb          5.2mb

Conditions:

4 Cores 2 NICs 16 GB RAM Automated Install Single Node rocknsm-20210514-14_14_30.iso

[spartan782]

This could be an issue with Elastic 7.11 it looks like. I will keep this open but want to try another elastic version to see if we get the same errors with the dashboards before changing them.

[webhead404]

As discussed I believe a good place to look for this issue is in the lighttpd proxy config. I confirmed this in 7.11 but didn't test 7.12. Here is a snippet of the lighttpd error log:

2021-03-11 16:10:38: (http-header-glue.c.1032) proxy backend sent invalid response header (Transfer-Encoding) to HTTP/1.0 request 2021-03-11 16:10:38: (gw_backend.c.2125) response not received, request sent: 927 on socket: tcp:127.0.0.1:5601 for /internal?, closing connection

[avladulescu]

Ended up removing the lighttpd which was buggy and installed nginx.

You can use this simple configuration as it works, but if your rock web access is in a closed network only. If you feel like switching to a SSL certificate and auth you will have a little bit more work to do.

        location / {
                proxy_pass http://127.0.0.1:5601;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_cache_bypass $http_upgrade;
                proxy_set_header    Host            $host;
                proxy_set_header    X-Real-IP       $remote_addr;
                proxy_set_header    X-Forwarded-for $remote_addr;
                port_in_redirect off;
                proxy_connect_timeout 3600;
            }

Also in kibana.yml add this so it doesn't buzz all the time with the publicbaseurl notice:

server.publicBaseUrl: "http://127.0.0.1:5601"
server.host: 127.0.0.1
server.port: 5601

Hope it helps.