bdon
Places tile creation failed for March 2025
During this month's places tiles run, we received a couple of errors and the job failed.
17
2025-03-19T21:51:02.549Z
'phones', phones,
18
2025-03-19T21:51:02.549Z
'brand', brand,
19
2025-03-19T21:51:02.549Z
'addresses', addresses,
20
2025-03-19T21:51:02.549Z
'version', version,
21
2025-03-19T21:51:02.549Z
'sources', sources
22
2025-03-19T21:51:02.549Z
) AS properties,
23
2025-03-19T21:51:02.549Z
row_number() over () as id,
24
2025-03-19T21:51:02.549Z
FROM read_parquet('$1/theme=places/type=place/*')
25
2025-03-19T21:51:02.549Z
) TO STDOUT (FORMAT json);
26
2025-03-19T21:51:02.549Z
"
27
2025-03-19T21:51:02.549Z
32 Aborted (core dumped) | tippecanoe -o $2 --force -J $SCRIPT_DIR/places.filter.json -l place -rg --drop-densest-as-needed --extend-zooms-if-still-dropping --maximum-tile-bytes=2500000 --progress-interval=10
28
2025-03-19T21:47:10.541Z
0.1% 0/0/0 0.2% 0/0/0 0.4% 0/0/0 0.5% 0/0/0 0.6% 0/0/0 0.7% 0/0/0 0.8% 0/0/0 0.9% 0/0/0 1.0% 0/0/0 1.1% 0/0/0 1.3% 0/0/0 1.5% 0/0/0 1.6% 0/0/0 1.8% 0/0/0 2.0% 0/0/0 2.1% 0/0/0 2.2% 0/0/0 2.4% 0/0/0 2.5% 0/0/0 2.6% 0/0/0 2.7% 0/0/0 2.9% 0/0/0 3.0% 0/0/0 3.1% 0/0/0 3.2% 0/0/0 3.4% 0/0/0 3.5% 0/0/0 3.6% 0/0/0 3.7% 0/0/0 3.9% 0/0/0 4.0% 0/0/0 4.1% 0/0/0 4.2% 0/0/0 4.3% 0/0/0 4.4% 0/0/0 4.5% 0/0/0 4.6% 0/0/0 4.7% 0/0/0 4.8% 0/0/0 4.9% 0/0/0 5.0% 0/0/0 5.1% 0/0/0 5.2% 0/0/0 5.3% 0/0/0 5.4% 0/0/0 5.5% 0/0/0 5.6% 0/0/0 5.7% 0/0/0 5.8% 0/0/0 5.9% 0/0/0 6.0% 0/0/0 6.1% 0/0/0 6.2% 0/0/0 6.3% 0/0/0 6.4% 0/0/0 terminate called after throwing an instance of 'std::bad_alloc'
29
2025-03-19T21:47:10.541Z
what(): std::bad_alloc
30
2025-03-19T21:16:11.259Z
Base zoom/drop rate: 0% Choosing a drop rate of -r1.206080 to keep 50000.000000 features in tile 13/6523/3849.
31
2025-03-19T21:16:11.259Z
Choosing a drop rate of -r1.756645 to keep 50000.000000 features in tile 12/3261/1924.
32
2025-03-19T21:16:06.151Z
Read 0.00 million features Read 0.30 million features Read 0.58 million features Read 0.83 million features Read 1.08 million features Read 1.37 million features Read 1.65 million features Read 1.93 million features Read 2.18 million features Read 2.42 million features Read 2.67 million features Read 2.92 million features Read 3.20 million features Read 3.44 million features Read 3.65 million features Read 3.95 million features Read 4.26 million features Read 4.54 million features Read 4.82 million features Read 5.06 million features Read 5.31 million features Read 5.53 million features Read 5.82 million features Read 6.12 million features Read 6.39 million features Read 6.67 million features Read 6.95 million features Read 7.25 million features Read 7.53 million features Read 7.78 million features Read 8.04 million features Read 8.30 million features Read 8.57 million features Read 8.86 million features Read 9.16 million features Read 9.45 million features Read 9.72 million features Read 10.00 million features Read 10.27 million features Read 10.54 million features Read 10.81 million features Read 11.03 million features Read 11.32 million features Read 11.58 million features Read 11.85 million features Read 12.14 million features Read 12.39 million features Read 12.60 million features Read 12.84 million features Read 13.09 million features Read 13.36 million features Read 13.63 million features Read 13.90 million features Read 14.18 million features Read 14.45 million features Read 14.72 million features Read 14.99 million features Read 15.27 million features Read 15.54 million features Read 15.81 million features Read 16.09 million features Read 16.36 million features Read 16.68 million features Read 16.97 million features Read 17.23 million features Read 17.49 million features Read 17.75 million features Read 18.02 million features Read 18.30 million features Read 18.57 million features Read 18.83 million features Read 19.10 million features Read 19.37 million features Read 19.64 million features Read 19.90 million features Read 20.17 million features Read 20.44 million features Read 20.72 million features Read 20.99 million features Read 21.26 million features Read 21.53 million features Read 21.81 million features Read 22.06 million features Read 22.31 million features Read 22.52 million features Read 22.79 million features Read 23.07 million features Read 23.32 million features Read 23.59 million features Read 23.86 million features Read 24.14 million features Read 24.41 million features Read 24.68 million features Read 24.96 million features Read 25.23 million features Read 25.50 million features Read 25.78 million features Read 26.05 million features Read 26.33 million features Read 26.60 million features Read 26.87 million features Read 27.12 million features Read 27.39 million features Read 27.64 million features Read 27.85 million features Read 28.11 million features Read 28.38 million features Read 28.65 million features Read 28.92 million features Read 29.19 million features Read 29.45 million features Read 29.69 million features Read 29.93 million features Read 30.18 million features Read 30.43 million features Read 30.69 million features Read 30.97 million features Read 31.25 million features Read 31.50 million features Read 31.76 million features Read 32.01 million features Read 32.30 million features Read 32.55 million features Read 32.80 million features Read 33.10 million features Read 33.42 million features Read 33.69 million features Read 33.93 million features Read 34.17 million features Read 34.42 million features Read 34.70 million features Read 34.98 million features Read 35.23 million features Read 35.48 million features Read 35.72 million features Read 35.99 million features Read 36.26 million features Read 36.53 million features Read 36.81 million features Read 37.08 million features Read 37.35 million features Read 37.63 million features Read 37.90 million features Read 38.18 million features Read 38.43 million features Read 38.66 million features Read 38.89 million features Read 39.18 million features Read 39.48 million features Read 39.76 million features Read 40.05 million features Read 40.30 million features Read 40.56 million features Read 40.83 million features Read 41.18 million features Read 41.53 million features Read 41.88 million features Read 42.23 million features Read 42.57 million features Read 42.92 million features Read 43.27 million features Read 43.62 million features Read 43.97 million features Read 44.32 million features Read 44.66 million features Read 45.00 million features Read 45.35 million features Read 45.70 million features Read 46.05 million features Read 46.38 million features Read 46.71 million features Read 47.04 million features Read 47.38 million features Read 47.71 million features Read 48.05 million features Read 48.38 million features Read 48.71 million features Read 49.04 million features Read 49.37 million features Read 49.70 million features Read 50.03 million features Read 50.36 million features Read 50.69 million features Read 51.02 million features Read 51.35 million features Read 51.68 million features Read 52.02 million features Read 52.34 million features Read 52.68 million features Read 53.02 million features Read 53.35 million features Read 53.68 million features Read 54.01 million features Read 54.35 million features Read 54.69 million features Read 55.03 million features Read 55.38 million features Read 55.72 million features Read 56.07 million features Read 56.41 million features Read 56.76 million features Read 57.11 million features Read 57.46 million features Read 57.81 million features Read 58.15 million features Read 58.49 million features Read 58.83 million features Read 59.17 million features Read 59.52 million features Read 59.86 million features Read 60.21 million features Read 60.56 million features Read 60.90 million features Read 61.25 million features Read 61.60 million features Read 61.95 million features Read 62.29 million features Read 62.64 million features Read 62.99 million features Read 63.33 million features Read 63.68 million features Read 64.02 million features Read 64.36 million features Read 64.70 million features Read 65.03 million features Read 65.36 million features Read 65.70 million features Read 66.03 million features Read 66.37 million features Merging string pool Merging vertices Merging nodes Merging index Reordering geometry: 0% Reordering geometry: 9% Reordering geometry: 24% Reordering geometry: 25% Reordering geometry: 26% Reordering geometry: 28% Reordering geometry: 32% Reordering geometry: 34% Reordering geometry: 37% Reordering geometry: 38% Reordering geometry: 43% Reordering geometry: 45% Reordering geometry: 48% Reordering geometry: 50% Reordering geometry: 51% Reordering geometry: 52% Reordering geometry: 56% Reordering geometry: 57% Reordering geometry: 58% Reordering geometry: 59% Reordering geometry: 71% Reordering geometry: 73% Reordering geometry: 74% Reordering geometry: 77% Reordering geometry: 78% Reordering geometry: 81% Reordering geometry: 83% Reordering geometry: 89% Reordering geometry: 91% Reordering geometry: 92% Reordering geometry: 93% Reordering geometry: 95% Reordering geometry: 96% Reordering geometry: 97% Reordering geometry: 98% Reordering geometry: 99% 66598099 features, 7861106253 bytes of geometry and attributes, 32536041539 bytes of string pool, 0 bytes of vertices, 0 bytes of nodes
33
2025-03-19T20:34:38.288Z
+ '[' places == admins ']'
34
2025-03-19T20:34:38.288Z
+ '[' places == places ']'
35
2025-03-19T20:34:38.288Z
+ test -d scripts/2025-03-19/places.sh
36
2025-03-19T20:34:38.288Z
+ bash scripts/2024-07-22/places.sh /data places.pmtiles
37
2025-03-19T20:34:38.206Z
download: s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places/type=place/part-00005-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet to data/theme=places/type=place/part-00005-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet
38
2025-03-19T20:34:33.882Z
download: s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places/type=place/part-00001-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet to data/theme=places/type=place/part-00001-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet
39
2025-03-19T20:34:33.816Z
download: s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places/type=place/part-00002-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet to data/theme=places/type=place/part-00002-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet
40
2025-03-19T20:34:33.316Z
download: s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places/type=place/part-00004-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet to data/theme=places/type=place/part-00004-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet
41
2025-03-19T20:34:32.863Z
download: s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places/type=place/part-00003-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet to data/theme=places/type=place/part-00003-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet
42
2025-03-19T20:34:32.010Z
download: s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places/type=place/part-00000-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet to data/theme=places/type=place/part-00000-b8709c80-62f4-4f9b-a3ff-c9d256bb6e6f-c000.zstd.parquet
43
2025-03-19T20:34:08.203Z
+ RELEASE_DATA=2025-03-19.0
44
2025-03-19T20:34:08.203Z
+ BUCKET=overturemaps-tiles-us-west-2-beta
45
2025-03-19T20:34:08.203Z
+ THEME=places
46
2025-03-19T20:34:08.203Z
+ OUTPUT_NAME=places
47
2025-03-19T20:34:08.203Z
+ SCRIPTS_VERSION=2024-07-22
48
2025-03-19T20:34:08.203Z
+ RELEASE_TILESET=2025-03-19
49
2025-03-19T20:34:08.203Z
+ aws s3 sync --no-progress --region us-west-2 --no-sign-request s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places /data/theme=places
The telltale issue seems to be the 2025-03-19T21:47:10.541Z what(): std::bad_alloc message, which seems to indicate that we ran out of memory. Is the places dataset much larger than it used to be?
Current limits for memory EC2 is 60GB [1]. Places currently use Tippecanoe for tile generation [2]. We can throw more memory at the job, but we may need to ultimately move to Planetiler as the size of places grows beyond 100GBs.
We went from 62 to 66 million places in the last release.
[1] https://github.com/OvertureMaps/overture-tiles/blob/main/overture-tiles-cdk/lib/overture-tiles-cdk-stack.ts#L121 [2] https://github.com/OvertureMaps/overture-tiles/blob/main/scripts/2024-07-22/places.sh#L32
Looks like we can bump the compute environment from using a c7gd.8xlarge (64GB) to c7gd.12xlarge (96GB) or 16xlarge (128GB) for temporary space buffer- I'm worried that might skyrocket our costs.
We could also adjust the script to try and do less processing, like say raise the zoom specified in places.filter.json from 14 to 15 and see if that helps.
Bumping the instance from a c7gd.8xlarge EC2 instance to a c7gd.16xlarge instance fixed the issue. This doubles the cost of the generation from ~$3 to ~6$ a month, so I'd say the extra buffer is worth the investment.
We could also consider moving to planetiler, but then we would lose the aggregated heat map view of places at higher zooms.
I think moving to a larger instance is the right temporary solution and should give us enough headroom in the medium term.
I've been prototyping a new memory-friendly generalization technique in planetiler that could let us drop the tippecanoe requirement and also get good overviews for buildings and addresses.
That sounds really great @bdon. It would be great to get the generalized views to convey where the coverage of the map is densest.
update: we ran into this issue again with the July release. Looks like part of the problem is the place density in several tiles, including null island. We attached an even larger compute environment (with these EC2 instances: m7gd.12xlarge, m7gd.16xlarge, r7gd.12xlarge, r7gd.16xlarge) to get the tile job to complete for this release. Definitely overkill but the job still took more than two hours. @bdon I'd love to hear more about your memory-friendly generalization technique in Planetiler.
I just ran the places tiles using c7gd.8xlarge on places 2025-08-20.0 and it succeeded in about 2h 30m, maybe something changed in the August release? It would be nice to reduce the runtime here, by compromising on the data in the lower zoom levels.