GNIP 102: Proposal to drop support for the deployment from the GeoNode core repo

[giohappy]

GNIP 102: Proposal to drop support for the deployment from the GeoNode core repo

Overview

The proposal is about dropping devops and infrastructure code for the direct deployment of GeoNode core. Switching to a GeoNode Project based approach is the general advice, and it would help maintenance.

Proposed By

giohappy - Giovanni Allegri ([email protected])

Assigned to Release

This proposal is for GeoNode .

State

• [x] Under Discussion
• [ ] In Progress
• [ ] Completed
• [ ] Rejected
• [ ] Deferred

Motivation

GeoNode can be deployed with two very different approaches:

• directly from the main repo (this one)
• as a dependency in a GeoNode Project

The general advice is to employ a GeoNode Project, for several reasons:

• it allows overriding GeoNode functionality (templates, tags, APIs, etc.) without modifications to the core modules
• it allows managing the instance as a standard Django project, which includes extending and customizing the included apps
• it allows easier upgrades between versions
• its deployment tooling (Docker, tasks, etc.) received more support since this is the most widely adopted approach

Maintaining two solutions is also becoming problematic in terms of maintenance and testing. We have duplicated code (docker files, tasks, .env sample files, etc.), with some small differences which also makes it easy to introduce errors. Maintaining the core code for deployment also requires having setup cfg data synced with requirements files, which are still the main solution for prepping the environment for a deployment.

Proposal

We propose to remove all code related to deployment operations from this repository, and just rely on GeoNode Project for serving GeoNode instances.

Backwards Compatibility

Projects that rely on deploying GeoNode from this repo will keep working, but they won't receive updates.

Future evolution

We might review the architecture of the GeoNode Project template to make it deployable wihtout having to generate a new project.

Feedback

None for the moment

Voting

Project Steering Committee:

• Alessio Fabiani:
• Francesco Bartoli:
• Giovanni Allegri:
• Toni Schoenbuchner:

[t-book]

Thanks @giohappy for this proposal. I see the benefits of removing redundant deployment code to reduce maintenance overhead. The only con I see is that using GeoNode project adds a little more complexity and that one loses the git history when generating the project...which might be challenging for new users. Anyways. We will still support both manual installation (advanced option in docs) and Docker installation via GeoNode Project, right?

[giohappy]

@t-book I understand your concerns. We also have similar doubts, and I'm still thinking about how to simplify and improve the maintenance of downstream projects. We're open to suggestions on this, and we could resume the discussion we had in the past on the topic, including the proposals from @ridoo

We will still support both manual installation (advanced option in docs) and Docker installation via GeoNode Project, right?

Well, you can install manually or with Docker in both cases.

[ridoo]

@giohappy nice to see more and more redundant tasks and configs are intended to be removed. GeoNode project currently is a place which has to be aligned to the changes made in core by moving and copying things. However, we had sevaral problems maintaining downstream projects over the last years. The problems I am talking about include:

• geonode-project is a template, from which you generate a custom project and detach from upstream. From the very beginning of your project, there is no shared git history to track future upstream changes (as @t-book already mentioned)
• building things based on dev or stable (master and .x lines) is cumbersome, as it takes very long (there is no appropriately tagged base images)

For those reasons (and some more convenient stuff making things more maintainable IMO) we created our own (opinionated) blueprint for geonode docker deployments which provides a trackable "template" which is easier™ to build and maintain. However, I'd love to see both approaches merge somehow. For that, I'd prefer a docker based scope of course, but I can see the (other's) need for non-docker deployments also. So if you have concrete plans how to go forward, I am sure we are able to put best of two worlds together.

[giohappy]

What about the following

• Remove anything related to deployment from this repo (.env.sample, fixtures, docker compose, etc.)
• Create a new geonode-platform repo, which will replace geonode-project to avoid confusion with the current setup. The repo will contain a slightly refactored instance generated from geonode-project, with the minimal files required to run a vanilla GeoNode.
• Customize and extend the geonode-platform with Django apps. Django already provides mechanisms to extend a project, so we might leverage this to customize projects, rather then instantiating new projects. Users can fork the geonode-platform, extend it as they want, and keep it aligned to the upstream repo.

@ridoo We can also rethink the Docker image setup. We might get back to having the Dockerfile inside the geonode repo, and just inherit it inside geonode-platform.

This is a rough idea. Maybe it won't be as easy as it seems, but it should work... I've started testing a minimal layout at https://github.com/giohappy/geonode-platform

[ridoo]

@giohappy nice!

The geonode-platform repo might be good to play around but I am hestitant to create yet another repo in the geonode universe ;). It should be fine to just communicate via an issue here. As users are not forking here anyway, this should be fine IMO (@t-book opinions?).

Some first comments:

• the GeoNode Dockerfile includes all build steps which makes building slow
• is it possible to move all build stuff to the GeoNode base image? The "project" repo then should just decide what to extend then
• with that, I am unsure about the fixtures .. are they really needed or do they just demonstrate how to add data?
• the .devcontainer config might not have to be generated
• do we want to maintain a create-env.py file? There is a samplt.env file which could be copied to (git-ignored) .env file. We should provide the GeoNode base image with reasonable defaults and only put in those settings a user should change (this is of course opinionated, but ok).
• is it really necessary to put the task.py here? What are the use cases to make adjustments here? Perhaps this would be better put in the GeoNode base image?

[giohappy]

I agree @ridoo another repo might be too much and confusing. We could keep using geonode-project but as a real project and not as a template. I've renamed it https://github.com/giohappy/geonode-project Before commenting on your points and any further topic, I want to emphasize that one of the goals of the project is to be able to run it without Docker, so its layout shouldn't be limited in that sense.

the GeoNode Dockerfile includes all build steps which makes building slow

Probably I miss the solution you have in mind. It should contain the GeoNode Project, ready to be started, which means that the image should be built from this repo. Maybe we could keep the master and development branches build from source, while tagged versions could replace the build phase with a reference to the Docker Hub image inside Docker compose.

BTW we could remove this line and move it the base image.

with that, I am unsure about the fixtures .. are they really needed or do they just demonstrate how to add data?

They preconfigure topic categories, the Geoserver Oauth2 app, the admin user, etc. Without fixtures users will have to create all these expected data manually. How would you avoid that?

the .devcontainer config might not have to be generated

Sure. This is just my own code that I reuse between projects. The .devcontainer files still contain references to the geonode project name, so I generate them to avoid having to adapt the configurations manually.

do we want to maintain a create-env.py file?

It's just a utility that saves time and avoids projects sharing the same secrets. In the past we have found many projects deployed with the default values!

is it really necessary to put the task.py here?

We must investigate if we can invoke the tasks using the module from GeoNode instead of the one bundled with the project. Can you do it @ridoo ?

[ridoo]

I mostly can speak from the Docker perspective TBH and will have to test things out first. I have some other topics in scope at the moment, but will get back to this one, in particular evaluating pros and cons compared to our approach embracing the pure Docker setup.

Update: A few comments regarding your response:

• We should agree on how a minimal project setup looks like and what GeoNode should bring into the project
• Personally, I would skip installing GeoNode from requirements.txt in a Docker based variant as a dependencies should be provided by the base image. This would boost the image creation as only custom dependencies would have to be installed (at least for Docker)
• Why are entrypoint.sh and tasks.py necessary to be part of the project actually? There are lots of GeoNode bootstrapping specifics and configuration. A project should not have that in scope, should it? Most of that is for GeoNode itself, so I'd leave it where it is actually used.
  • The actual main point I see is, that tasks.py, create-env.py, fixtures/ and entrypoint.sh mixes a lot of configuration which may be valid for Docker only or seem to be overly complex to accomplish. Examples are oauth or admin config which exist as fixture, but are actually generated from env variables and loaded as temporary fixtures in the tasks.py
  • I see that tasks in tasks.py are only invoked from the Docker's entrypoint.sh but we should focus on only one configuration variant
• one general goal I had in mind when creating the docker-blueprint was to "hide" as many as possible and provide a reasonable default .. at the same time providing locations where to re-configure and adjust things. Of course, this will not remove complexity (it's still there), but I wanted to reduce the risk to break things because of fiddling around settings (some of them are brittle to be set in combination) and leave it to the experienced user to configure the system by picking only those settings which are necessary