Feature Request: Import / Export configuration

[kvaps]

Feature request

It would be nice for have an opportunity for export current controller configuration in some machine readable format like json, yaml, or just simple commands.

Yes I know that I can use -m flag for save some resources partially. But there is no way for export all configuration.

The is also no opportunity for upload it back, fully and partially.

Goals

• Simple backup / restore of configuration.
• Simple migration configuration from one server to another.
• Independence from database type.
• Simplicity for upload initial configuration to server during deploy.
• Better for automation and integration with 3rd party software.

Interface changes

We can add new command like linstor configuration:

Something like that:

usage: linstor configuration [-h]
                    {export, import} ...

Configuration subcommands

optional arguments:
  -h, --help            show this help message and exit

Node commands:
   - export (e)
   - import (i)

  {export, import}

usage: linstor configuration export [-h]
                           [--format {commands,json}]
                           [--scope {controller, encryption, node, resource,
                                   resource-definition, snapshot, storage-pool,
                                   storage-pool-definition, volume-definition} ...]

Exports configuration from linstor cluster.

optional arguments:
  -h, --help            show this help message and exit
  --format {commands,json}
                        Export format (default: json)
  --scope   {controller, encryption, node, resource,
            resource-definition, snapshot, storage-pool,
            storage-pool-definition, volume-definition} ...
                        Scope for export (default: all)

usage: linstor configuration import [-h]
                           [--format {commands,json}]
                           [--overwrite {yes,no}]
                           filename

Imports configuration to linstor cluster.

optional arguments:
  -h, --help            show this help message and exit
  --format {json}
                        Import format (default: json)
  --overwrite {yes,no}
                        Overwrite existing parameters (default: yes)

positional arguments:
  filename              File, contain exported configuration.

Maybe have you something already in controller debug console? It is really wanted feature for me.

[raltnoeder]

There are currently no plans for adding such a feature in the near future, since that would be an excessively complex task to implement, without adding much useful functionality.

Regarding the goals:

• Backup/restore is supposed to be handled by the database
• Migration or duplication of the configuration can already be handled by moving or duplicating the database
• Initial configuration can be scripted, or in very rare cases where that would make sense, loaded into the database directly, however, this method is discouraged
• Automatic migration between different types of databases is currently not supported
• 3rd party software should never change the entire configuration at once by importing a dump, but should use the network API to interact with the controller in a transaction-safe and clean way

[elonen]

Even if there's no backup dump in LINSTOR, some kind of support for somehow gracefully resyncing / recovering from a (slightly) outdated database is practically quite necessary. I've currently got a LINSTOR controller that was corrupted during a cluster crash and needed to be restored from backups. Now /var/lib/linstor.d (and LVM) contains resources that aren't shown by linstor r list, and therefore can't be deleted by linstor rd delete.

Perhaps a command like linstor resource-definition import <resource.res>?

[kvaps]

I've prepared small shell script for export configuration from linstor-server as simple commands for myself. https://github.com/kvaps/linstor-backup-script I hope it will be useful for someone.

Note: it does not exports all the things, just basic, so it does not replace normal backups. If you want something more, PRs are welcome :)

[rp-]

Now /var/lib/linstor.d (and LVM) contains resources that aren't shown by linstor r list, and therefore can't be deleted by linstor rd delete.

If you restart the satellite, it will delete all res files in /var/lib/linstor.d and only populate those registered on the controller.

some kind of support for somehow gracefully resyncing / recovering from a (slightly) outdated database is practically quite necessary.

I don't really get what you mean by outdated database?

[raltnoeder]

Importing DRBD resource files is being considered as a future feature. Importing existing resources manually is currently not totally trivial, but possible - see https://www.linbit.com/en/how-migrating-manually-created-resources-linstor/

[kvaps]

Right now I was successful migrated linstor from h2 to postgress backend, it wasn't so easy.

1. I made copy of local H2 database
2. Then I initializated new linstor controller with postgress backend, and stopped it
3. After that I used SQuirreL SQL Client and conneted to both databases.
4. Selected all tables on LINSTOR/TABLES except FLYWAY_SCHEMA_HISTORY (database should be same version) then Scripts --> Create Data Script It will generate big SQL script with a lot's of INSERT
5. Then you can import it into postgress, but you need to lowercase table names before:

sed  -i 's|"LINSTOR"\."\([A-Z_]\+\)"|"LINSTOR"\."\L\1"|g' dump.sql

7. Note: some tables should be processed before some others, eg stor_pool_definitions should be created before resources

[garrmark]

+1 for an easy way to migrate from the h2 database to a different database