Adding other extension parameters to .env file / templates for extensions

[terchris]

Hi I’d like to use docker-ckan to create a setup that can run as a docker container on Azure / Google cloud. To avoid saving files to containers and docker host VM I would like to use the https://github.com/TkTech/ckanext-cloudstorage

The ckanext-cloudstorage has some parameters that must go in the production.ini file: ckanext.cloudstorage.driver = AZURE_BLOBS ckanext.cloudstorage.container_name = demo ckanext.cloudstorage.driver_options = {"key": "", "secret": ""}

I like the way you use .env to set parameters for production.ini and would like to put the ckanext-cloudstorage parameters in .env file as well.

I’m thinking about the following solution and would like to know what you think.

In the .env file CKANEXT__CLOUDSTORAGE__DRIVER = AZURE_BLOBS CKANEXT__CLOUDSTORAGE__CONTAINER_NAME = <your ...> CKANEXT__CLOUDSTORAGE__DRIVER_OPTIONS_PUBLIC_KEY = CKANEXT__CLOUDSTORAGE__DRIVER_OPTIONS_SECRET_KEY =

I see that in the development setup there is a start_ckan_development.sh that execute files in the "/docker-entrypoint.d" dir.

If there was a similar setup for the production setup. Then we could create several template files in "/docker-entrypoint.d" dir. In my case a file named ckanext-cloudstorage.template The ckanext-cloudstorage.template looking like this:

RUN paster --plugin=ckan config-tool ${CKAN_INI} "ckanext.cloudstorage.driver = ${CKANEXT__CLOUDSTORAGE__DRIVER}" && \ paster --plugin=ckan config-tool ${CKAN_INI} "ckanext.cloudstorage.container_name = ${CKANEXT__CLOUDSTORAGE__CONTAINER_NAME}" && \ paster --plugin=ckan config-tool ${CKAN_INI} "ckanext.cloudstorage.container_name = ${CKANEXT__CLOUDSTORAGE__CONTAINER_NAME}" && \ paster --plugin=ckan config-tool ${CKAN_INI} "ckanext.cloudstorage.driver_options = {"key": "${CKANEXT__CLOUDSTORAGE__DRIVER_OPTIONS_PUBLIC_KEY}", "secret": "${CKANEXT__CLOUDSTORAGE__DRIVER_OPTIONS_SECRET_KEY}" } "

So when I would like to install ckanext-cloudstorage I go to the "/docker-entrypoint.d" dir and copy the ckanext-cloudstorage.template to ckanext-cloudstorage.sh

The ckanext-cloudstorage.sh executes on sartup. Sets the parameters i the production.ini. Reloads ckan and I have a working extension.

What do you think about the idea and how do you see it can be done?

Regards Terje

[terchris]

In order to reduce files that needs to be edited I think that the installation of the extension should be moved into the template file. Eg move the pip install -e git+https://github.com/TkTech/ckanext-cloudstorage.git#egg=ckanext-cloudstorage From the Dockerfile to the ckanext-cloudstorage.template file.

In the ckanext-cloudstorage.template we can also add the paster --plugin=ckan config-tool $CKAN_INI "ckan.plugins = cloudstorage"

If there is a need to control the load sequence of the extensions we could add a number in fromt of the template file and execute the templates in alphabetic order. eg rename ckanext-cloudstorage.template to 1ckanext-cloudstorage.sh

we could also add howto text/version/author in the template.

Terje

[gerbyzation]

Hi @terchris,

I'm not sure what exactly you're hoping to improve with templates, could you clarify?

If it is to set the config for the extension, there is no need to run the paster tool when using environment variables like you suggested doing. They will be loaded into the config object through ckanext-envvars when CKAN starts, they don't need to be declared in the config file as well.

If it is to be able to dynamically load plugins without rebuilding the image, I think that's not a great idea for production usage. For production it is generally better to have ready-build containers with the required dependencies. It'll slow down startup time significantly (minutes depending on dependencies) and probably make your setup harder to test and troubleshoot. Also AFAIK paster --plugin=ckan config-tool $CKAN_INI "ckan.plugins = cloudstorage" would be problematic as it would unset any other plugins previously set. For this to not discard previously set plugins the list of installed plugins would need to be retrieved and the new plugin added to this list.

I'm running images based of this repo and using cloudstorage as well, deploying on kubernetes. The extension gets installed in the Dockerfile (ckan/Dockerfile) and configured through environment variables, so I can run several CKAN instances with different configurations off the same image.