Running out of space when downloading the planet

[Shadow-shadown]

I'm using this docker to download the planet My system configuration is 10 core cpu and 128gb ram and 2.2TB storage

I'm running into the issue of no space in disk when I check it it shows 57% used

The error

• chown -R renderer: /home/renderer/src/ /data/style/
• '[' -f /data/style/scripts/get-external-data.py ']'
• '[' -f /data/style/external-data.yml ']'
• sudo -E -u renderer python3 /data/style/scripts/get-external-data.py -c /data/style/external-data.yml -D /data/style/data INFO:root:Starting load of external data into database Traceback (most recent call last): File "/data/style/scripts/get-external-data.py", line 311, in main() File "/data/style/scripts/get-external-data.py", line 182, in main os.makedirs(data_dir, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 28] No space left on device: '/data/style/data'

The docker command used to run the container

docker run -d --shm-size="10G" -e DOWNLOAD_PBF=https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf -e PGPASSWORD=${PGPASSWORD} -e "OSM2PGSQL_EXTRA_ARGS=-C 40096" -e "FLAT_NODES=enabled" -v /osm-data:/data/database/ -v osm-tiles:/data/tiles/ tileserver import

The disk space

sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 99G 0 part ├─oar_ubuntu-swap 253:1 0 3.9G 0 lvm [SWAP] ├─oar_ubuntu-home 253:2 0 9.8G 0 lvm /home └─oar_ubuntu-root 253:4 0 85.3G 0 lvm / sdb 8:16 0 2.7T 0 disk └─sdb1 8:17 0 2.7T 0 part ├─data-osm--data 253:0 0 2.1T 0 lvm /osm-data └─data-osm--tiles 253:3 0 500G 0 lvm /osm-tiles

What am I doing wrong could someone help me out

[Istador]

docker run -d --shm-size="10G" -e DOWNLOAD_PBF=https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf -e PGPASSWORD=${PGPASSWORD} -e "OSM2PGSQL_EXTRA_ARGS=-C 40096" -e "FLAT_NODES=enabled" -v /osm-data:/data/database/ -v osm-tiles:/data/tiles/ tileserver import

NOT an explanation for the error, but -v osm-tiles:/data/tiles/ should be -v /osm-tiles:/data/tiles/ instead, if you want it saving to the extra partition that you created and that is mounted to /osm-tiles. Otherwise it saves to /var/lib/docker/volumes/....

(Unless you created the osm-tiles named volume manually with providing a mount point, but because you didn't seem to do that for /osm-data I assume you made a mistake here.)

---

Interestingly it says:

No space left on device: '/data/style/data'

/data/style/data is not part of the mounted volumes for /data/database/ or /data/tiles/ and is therefore part of the containers internal storage.

So the issue seems to be related to the root partition oar_ubuntu-root that holds the containers (/var/lib/docker/containers/...). Seemingly it doesn't have enough space to hold some temporary files downloaded in a later stage of the import (the main import of the planet.osm is already done).

Note: the ~75 GB planet-latest.osm.pbf was also downloaded into the container to /data/region.osm.pbf and resides on the root partition.

Note: you don't execute docker run with the --rm parameter, so it doesn't automatically delete containers and their internal storage automatically, so you need to do that manually.

---

An improvement to this image to prevent this error would be to add the following code to the run.sh after the osm2pgsql step of the import to clean up the temporarily downloaded files after the main import step:

# clean up downloaded files
if [ -n "${DOWNLOAD_PBF:-}" ]; then
    rm /data/region.osm.pbf
    if [ -n "${DOWNLOAD_POLY:-}" ]; then
        rm /data/region.poly
    fi
fi

[Shadow-shadown]

Is there anyway that I can mount it to a volume or the temporary remove commands on top will be a better solution ..?

[Istador]

You could download the file before and then mount it as described in the readme.

    -v /absolute/path/to/luxembourg.osm.pbf:/data/region.osm.pbf \

Otherwise you'd need to modify the run.sh yourself and build a new image.

I created PR #420 to fix this, but I don't believe it will be merged anytime soon. The last update to this project was 10 months ago.