gravitystorm
Adapt Docker setup to latest changes
Fixes #5031 Fixes #5005
Requires #4893, or any other solution for #5043.
The Docker setup is a quick and easy way to get an Openstreetmap-carto dev environment running, so it’s probably worth keeping it in a working state. This PR updates the startup script to take recent changes into account, such as #4978. It also updates the base images to more recent versions.
With this PR alone, setting up the containers fails because fonts cannot be downloaded. Applying #4893 allows testing the whole setup.
Many thanks for working on this. Some minor comments from giving it a test run:
docker-compose up import to import the data (only necessary the first time or when you change the data file).
It would be useful to add that that downloading and importing the water polygons table can take substantial time, and will be the rate limiting step when importing a small subset of OSM data (which will be the normal use case). It doesn't help that the message shown by get-external-data.py is "Checking table X" during download, which gives the impression that things have got stuck (why is "checking" taking so long?).
On the first run, import failed with "Max retries exceeded with url: /download/antarctica-icesheet-outlines-3857.zip (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f345affb7d0>: Failed to resolve 'osmdata.openstreetmap.de", which is odd because the website was functioning. It worked on repeating the docker-compose up import (again this seems to be more an issue/fragility of get-external-data.py rather than the docker set up per se).
[Haven't tested beyond the font import step. This fails with "scripts/get-fonts.sh: 11: curl: not found".]
It doesn't help that the message shown by get-external-data.py is "Checking table X" during download, which gives the impression that things have got stuck (why is "checking" taking so long?).
I have added a few more log message to help with tracking the import process, and mentioned the external data download in DOCKER.md.
The latest version can now go through the import process without having to apply more patches. Noto Emoji is downloaded from the Internet Archive using the file @pjduplooygis has uploaded earlier this year. And because fonts are downloaded separately during the import process, they have been removed from the package list in the Dockerfile.
I don’t really want to go much further regarding fonts, as it isn’t the main topic of this PR.
Have now confirmed this works when merged with #4893.
The (existing) instructions about setting environment variables e.g. to adjust the behaviour of get-external-data.py are a bit confusing and don't seem to apply to running Docker containers from Windows/WSL. But that is a separate topic to this PR, which is focussed on getting the Docker set-up working again.
I'm familiar with docker as a user but wouldn't describe myself as an expert.
The updated Dockerfile looks good to me. It works (once the get-fonts issue is addressed) and is based on a long-term support version of Ubuntu.
It's a clear step forward from the current file, which is broken. That said, it might need tweaking depending on which solution to the font issue is deployed.
Fixes #5031, #5005.
@tpetillon I think you need two Fixes # lines for it to correctly link to both issues (it currently only links to #5031).
Rebased and re-tested.
Thanks for the reviews, and unblocking the font situation.
I think this is ready to be pulled in. @tpetillon is using it for development, and so it's clearly a significant advance on the current broken situation.
We currently have a bunch of issues marked as "good first issue" (I think that #5070 falls under this category rather than "help wanted"?). But it's quite a barrier for new contributors to set up a full rendering chain. Running kosmtik under Docker will be more accessible for many.
Thanks for the review. That would make this ready for merging. Are there any comments if #5125 is preferrable over this? Otherwise i would merge this and any improvements from #5125 not in here would need to go as changes on top of this.
I see no blockers with proceeding with this PR. The difference between this and #5125 are not significant. If there are any PRs which depend on a newer version of osm2pgsql than is in the ubuntu:noble repo, we would probably need to change the docker config to compile that from source, but that is not a major change.
Short version: recommend proceeding with this, and closing #5125 as OBE.
We are currently under-documented w.r.t. PostgreSQL/PostGIS version requirements and i opened #5129 about this.
Postgresql 17 could be useful for test environments because of improved analytic features (EXPLAIN).