[grimoirelab] Migrating .kibana from ES/Kibiter 6.1 to 6.8

[valeriocos]

The upcoming release 0.2.41 will support ES/Kibiter 6.8. In case your docker-compose is using volumes and you plan to update your environment, you need to execute the 2 scripts available here.

These scripts allow to migrate the information included in the .kibana index (which contains all objects used by Kibiter such as dashboards and index patterns) to ES/Kibiter 6.8. This is needed since (due to a change in upstream) the .kibana mapping in 6.8 is set to strict, which isn't compatible with the previous .kibana mapping used in GrimoireLab (i.e., dynamic).

Create .grimoirelab-sigils index The first script to execute is createsigilsindex.py. It identifies the dashboards and index patterns in the current .kibana, and generates the .grimoirelab-sigils index. This index contains the ID of the standard index patterns/dashboards together with their release date.

To execute the script you need to set the target ES (variable ENV) together with the credentials to access it (variables USER and PWD).

Migrate the .kibana The second script to execute is migration68.py. It does the following:

1. create a local directory in your machine at FOLDER_PATH
2. dump in the directory the mappings and data from a target Kibana instance
3. process the files downloaded to remove the release_date and set the mappings to strict
4. print a set of commands to:
   • a) delete the .kibana index
   • b) upload the new mappings and data to the index .kibana_old in the target Kibana instance
   • c) set the alias .kibana to the index .kibana_old
   • d) check that the alias is set correctly

To execute the script you need to set the target ES (variable ENV), the credentials to access it (variables USER and PWD) and the local path where to store and modify the .kibana index.

The workflow is the following:

• run the script
• check that the mappings and data have been correctly downloaded in FOLDER_PATH
• delete the current .kibana with 4a
• upload the .kibana_old generated with 4b
• set the alias .kibana to .kibana_old with 4c
• check that the alias is set correctly with 4d
• stop the docker-compose
• update and restart the docker-compose (see docker-compose files at https://github.com/chaoss/grimoirelab/tree/master/docker-compose)

[sduenas]

I think it would be useful to add this to the README file.

[marcofranssen]

@valeriocos When is this release to be expected?

Are those 2 scripts supposed to be ran only when upgrading to this 0.2.41 release?

Specifically asking for the Docker image grimoirelab/installed.

[valeriocos]

@valeriocos When is this release to be expected?

It's already out (currently we are in 0.2.42)

Are those 2 scripts supposed to be ran only when upgrading to this 0.2.41 release?

No, they should be run when passing from ES/Kibiter 6.1 to ES/Kibiter 6.8

Specifically asking for the Docker image grimoirelab/installed.

The Docker image grimoirelab/installed isn't up-to-date (cc @jgbarah). I would suggest to shift to the docker-compose solution in the meanwhile (https://github.com/chaoss/grimoirelab/tree/master/docker-compose)

[marcofranssen]

Should we use the mordred container instead of the grimoirelab:installed container? What functionality will be lost if we swap? Which containers is mostly updated?