Automatically load kibana dashboards

[erinreeves]

Summary

As a	metadata manager
I want to	have dashboards loaded automatically in kibana
So that I can	use the dashboards

Tasks

• [ ] Automatically load kibana dashboards upon the container starting up.

[erinreeves]

Few options: If we loaded the kibana dashboard via the API we'd still need a script then to automatically get kicked off, and expose that port (which seems a bit much simply for loading initial data). Oher thought is to copy the file over to the container and you could either do the above but do the script from within the container or have kibana configured to automatically load the file when kibana is ready to do so.

Evan: have kibana configured to automatically load the file when kibana is ready to do so my guess is that that’s your easiest path based on where it sounds like you are now

[erinreeves]

created branch 1297-kibana-dashboards

[erinreeves]

Kibana info so as to load the dashboard json file.

1. we running ECK WHICH IS AN OPERATOR
2. its going to be different from the conventional one (example under helm/onestop-registry)

[erinreeves]

To accomplish this would need to upload a file via kubernetes to the kibana container and put the path to this file in the metricbeat.yaml under the sections with the dashboards enabled section:

  dashboards:
    enabled: true
    file:

#or do url:

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html

[erinreeves]

I seem to not have some containers up, one of them being the filebeat one which is where the config adjustment to turn on dashboards auto loading is

NAME READY STATUS RESTARTS AGE onestop-client-cbfbc4668-vrzqn 1/1 Running 0 15h onestop-dev-cp-control-center-7cc7cb97d8-8h692 1/1 Running 0 15h onestop-dev-cp-kafka-0 2/2 Running 0 15h onestop-dev-cp-schema-registry-6b8859c769-8r8hq 2/2 Running 2 15h onestop-dev-cp-zookeeper-0 2/2 Running 0 15h onestop-dev-es-default-0 1/1 Running 0 15h onestop-dev-ingress-nginx-controller-5f9c9d7569-9ppkz 1/1 Running 0 15h onestop-dev-kb-6c9f7c8bdb-xzm5n 1/1 Running 0 15h onestop-indexer-779c5b57cf-89sqt 2/2 Running 4 15h onestop-manager-794cbb5d8c-rq4r2 2/2 Running 1 15h onestop-registry-0 2/2 Running 5 15h onestop-search-6564c8b9d-t8dsj 2/2 Running 2 15h

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html#url-option

[erinreeves]

semere:

1. use one off script to upload via the api or something else
2. based on ECK doc, you can upload by pointing to a url (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-dashboards.html)
3. or may be you can mount the file to the kibana pod default path or even custom path to upload the file by enabling the configuration
4. other approach is using filebeat (https://www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html), which is what @erin.reeves wanted to try

[erinreeves]

Tried loading kibana dashboards via changing the configs in metricbeat as well as filebeat with no luck. Config setting tried: dashboards: enabled: true url: "https://github.com/cedardevs/onestop/blob/1297-load-kibana-dashboards/elastic-common/src/main/resources/dashboards/errors.zip"

Always get this error regardless of file format: 2020-07-23T20:18:43.227Z ERROR instance/beat.go:933 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing URL/file: Failed to unzip the archive: /tmp/tmp536853400/errors.zip: zip: not a valid zip file Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing URL/file: Failed to unzip the archive: /tmp/tmp536853400/errors.zip: zip: not a valid zip file

Tried several file formats. errors.zip (created via zip errors.zip errors.ndjson) errors.ndjson.zip (created via right clicking the file on Mac in file finder and selecting compress errors.ndjson) errors errors.jdjson (think was via gunzip, was first attempt and didn't work)

[erinreeves]

Found I was using the url to viewing the file in the pretty view in github, not the raw file. Changed the url to the raw file. Tried the: .ndjson - zip: not a valid zip file .zip - Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing URL/file: Too many directories under /tmp/tmp110104288 .ndjson.zip - same, too many directories.