pygmaps-ng
pygmaps-ng copied to clipboard
Improve speed, especially testing polygon simplification
Creating a 250 MB map takes around half an hour (? I have no idea) on my little netbook. A 3 MB file takes ~270 seconds. This makes it difficult to test different color schemes and threshold/precision numbers.
The culprit is polygon simplification, which is enabled regardless of if it gets used. Once built, the simplifier can return polygons of different compressions very fast. So:
- Perhaps do not build simplifier if it is not used.
- If simplifier is built, allow the dataset to have it's precision modified and spit out new polygons on every call to Map.build_page.
- Along with 2 above, let colors be modified also, including colors inherited from the dataset.
cProfile output for plot_nature.py, which creates a 3 MB map from a .shp file:
53865448 function calls (53862141 primitive calls) in 272.681 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.148 0.148 272.689 272.689 plot_nature.py:3(<module>)
1200 0.914 0.001 165.183 0.138 pygmaps_ng.py:223(add_polygon)
3500 0.059 0.000 163.015 0.047 polysimplify.py:63(__init__)
3500 53.008 0.015 160.674 0.046 polysimplify.py:71(build_thresholds)
1200 0.009 0.000 79.262 0.066 plot_nature.py:25(safe_add_poly)
3346 0.045 0.000 78.833 0.024 geometries.py:640(tuple)
3500 5.123 0.001 78.193 0.022 geometries.py:583(tuple)
2262387 32.805 0.000 73.029 0.000 geometries.py:559(__getitem__)
6720204 52.718 0.000 61.121 0.000 polysimplify.py:41(triangle_area)
50 0.016 0.000 50.801 1.016 geometries.py:699(tuple)
2158193 6.237 0.000 23.867 0.000 fromnumeric.py:935(argmin)
1 0.141 0.141 19.871 19.871 pygmaps_ng.py:94(build_page)
1 0.000 0.000 19.331 19.331 __init__.py:43(jsmin)
1 9.598 9.598 19.328 19.328 __init__.py:72(minify)
2262387 11.881 0.000 18.073 0.000 geometries.py:199(_get_coord_dim)
2158193 17.631 0.000 17.631 0.000 {method 'argmin' of 'numpy.ndarray' objects}
2258887 16.468 0.000 16.468 0.000 polysimplify.py:51(remove)
2265887 8.375 0.000 11.494 0.000 geometries.py:224(point_count)
6894254 9.154 0.000 9.154 0.000 base.py:18(_get_ptr)
6720796 8.406 0.000 8.406 0.000 {abs}
6803962 7.775 0.000 7.775 0.000 {_ctypes.byref}
1 0.005 0.005 7.352 7.352 GIS.py:53(get_data_from_shp)
2258902 5.670 0.000 5.670 0.000 {method 'pop' of 'list' objects}
3250230 4.053 0.000 4.053 0.000 {method 'read' of 'cStringIO.StringI' objects}
2586010 3.677 0.000 3.678 0.000 {isinstance}
1200 3.521 0.003 3.531 0.003 geometries.py:384(transform)
12 0.083 0.007 3.512 0.293 layer.py:190(get_fields)
768151 2.019 0.000 3.101 0.000 __init__.py:79(write)
2493787 2.323 0.000 2.323 0.000 {method 'append' of 'list' objects}
15613 1.689 0.000 1.959 0.000 layer.py:59(__iter__)
14400 0.107 0.000 1.534 0.000 feature.py:110(get)
3519 1.470 0.000 1.470 0.000 {sorted}
12635 1.247 0.000 1.247 0.000 {numpy.core.multiarray.array}
768151 1.082 0.000 1.082 0.000 {method 'write' of 'cStringIO.StringO' objects}
179582 0.495 0.000 1.067 0.000 pygmaps_ng.py:67(saferound)
14400 0.102 0.000 1.000 0.000 feature.py:38(__getitem__)
2 0.002 0.001 0.874 0.437 __init__.py:24(<module>)
179582 0.572 0.000 0.572 0.000 {round}
3500 0.102 0.000 0.545 0.000 geometries.py:626(__getitem__)
14400 0.196 0.000 0.505 0.000 field.py:20(__init__)
1 0.001 0.001 0.477 0.477 pygmaps_ng.py:3(<module>)
1 0.002 0.002 0.394 0.394 matplottools.py:3(<module>)
6896 0.097 0.000 0.374 0.000 geometries.py:74(__init__)
14400 0.105 0.000 0.372 0.000 feature.py:119(index)
187144/187135 0.352 0.000 0.355 0.000 {method 'join' of 'str' objects}
2196 0.073 0.000 0.354 0.000 geometries.py:664(__getitem__)
42598 0.187 0.000 0.313 0.000 base.py:25(_set_ptr)
1 0.001 0.001 0.296 0.296 polysimplify.py:35(<module>)
5697 0.117 0.000 0.292 0.000 geometries.py:268(_get_srs)
1 0.002 0.002 0.282 0.282 pyplot.py:17(<module>)
1 0.007 0.007 0.270 0.270 layer.py:199(get_geoms)
14405 0.122 0.000 0.246 0.000 encoding.py:139(force_bytes)
4800 0.014 0.000 0.228 0.000 field.py:94(value)
263 0.003 0.000 0.220 0.001 re.py:230(_compile)
20 0.015 0.001 0.217 0.011 __init__.py:1(<module>)
125 0.002 0.000 0.216 0.002 sre_compile.py:567(compile)
4800 0.051 0.000 0.214 0.000 field.py:56(as_string)
121 0.001 0.000 0.214 0.002 re.py:192(compile)
12 0.213 0.018 0.213 0.018 {open}
15600 0.056 0.000 0.177 0.000 feature.py:25(__init__)
1 0.002 0.002 0.174 0.174 colorbar.py:20(<module>)
30159 0.128 0.000 0.167 0.000 errcheck.py:111(check_pointer)
1 0.000 0.000 0.140 0.140 __init__.py:33(<module>)
125 0.002 0.000 0.133 0.001 sre_parse.py:706(parse)
4956 0.039 0.000 0.132 0.000 encoding.py:84(force_text)
1 0.000 0.000 0.130 0.130 color_gen.py:1(<module>)
1 0.000 0.000 0.130 0.130 __init__.py:6(<module>)
1 0.002 0.002 0.129 0.129 brewer2mpl.py:1(<module>)
355/125 0.005 0.000 0.128 0.001 sre_parse.py:317(_parse_sub)
6896 0.044 0.000 0.127 0.000 geometries.py:239(geom_type)
479/135 0.033 0.000 0.127 0.001 sre_parse.py:395(_parse)
1 0.000 0.000 0.126 0.126 driver.py:2(<module>)
1 0.003 0.003 0.125 0.125 ds.py:5(<module>)