pygmaps-ng icon indicating copy to clipboard operation
pygmaps-ng copied to clipboard

Improve speed, especially testing polygon simplification

Open Permafacture opened this issue 10 years ago • 0 comments

Creating a 250 MB map takes around half an hour (? I have no idea) on my little netbook. A 3 MB file takes ~270 seconds. This makes it difficult to test different color schemes and threshold/precision numbers.

The culprit is polygon simplification, which is enabled regardless of if it gets used. Once built, the simplifier can return polygons of different compressions very fast. So:

  1. Perhaps do not build simplifier if it is not used.
  2. If simplifier is built, allow the dataset to have it's precision modified and spit out new polygons on every call to Map.build_page.
  3. Along with 2 above, let colors be modified also, including colors inherited from the dataset.
cProfile output for plot_nature.py, which creates a 3 MB map from a .shp file: 

         53865448 function calls (53862141 primitive calls) in 272.681 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.148    0.148  272.689  272.689 plot_nature.py:3(<module>)
     1200    0.914    0.001  165.183    0.138 pygmaps_ng.py:223(add_polygon)
     3500    0.059    0.000  163.015    0.047 polysimplify.py:63(__init__)
     3500   53.008    0.015  160.674    0.046 polysimplify.py:71(build_thresholds)
     1200    0.009    0.000   79.262    0.066 plot_nature.py:25(safe_add_poly)
     3346    0.045    0.000   78.833    0.024 geometries.py:640(tuple)
     3500    5.123    0.001   78.193    0.022 geometries.py:583(tuple)
  2262387   32.805    0.000   73.029    0.000 geometries.py:559(__getitem__)
  6720204   52.718    0.000   61.121    0.000 polysimplify.py:41(triangle_area)
       50    0.016    0.000   50.801    1.016 geometries.py:699(tuple)
  2158193    6.237    0.000   23.867    0.000 fromnumeric.py:935(argmin)
        1    0.141    0.141   19.871   19.871 pygmaps_ng.py:94(build_page)
        1    0.000    0.000   19.331   19.331 __init__.py:43(jsmin)
        1    9.598    9.598   19.328   19.328 __init__.py:72(minify)
  2262387   11.881    0.000   18.073    0.000 geometries.py:199(_get_coord_dim)
  2158193   17.631    0.000   17.631    0.000 {method 'argmin' of 'numpy.ndarray' objects}
  2258887   16.468    0.000   16.468    0.000 polysimplify.py:51(remove)
  2265887    8.375    0.000   11.494    0.000 geometries.py:224(point_count)
  6894254    9.154    0.000    9.154    0.000 base.py:18(_get_ptr)
  6720796    8.406    0.000    8.406    0.000 {abs}
  6803962    7.775    0.000    7.775    0.000 {_ctypes.byref}
        1    0.005    0.005    7.352    7.352 GIS.py:53(get_data_from_shp)
  2258902    5.670    0.000    5.670    0.000 {method 'pop' of 'list' objects}
  3250230    4.053    0.000    4.053    0.000 {method 'read' of 'cStringIO.StringI' objects}
  2586010    3.677    0.000    3.678    0.000 {isinstance}
     1200    3.521    0.003    3.531    0.003 geometries.py:384(transform)
       12    0.083    0.007    3.512    0.293 layer.py:190(get_fields)
   768151    2.019    0.000    3.101    0.000 __init__.py:79(write)
  2493787    2.323    0.000    2.323    0.000 {method 'append' of 'list' objects}
    15613    1.689    0.000    1.959    0.000 layer.py:59(__iter__)
    14400    0.107    0.000    1.534    0.000 feature.py:110(get)
     3519    1.470    0.000    1.470    0.000 {sorted}
    12635    1.247    0.000    1.247    0.000 {numpy.core.multiarray.array}
   768151    1.082    0.000    1.082    0.000 {method 'write' of 'cStringIO.StringO' objects}
   179582    0.495    0.000    1.067    0.000 pygmaps_ng.py:67(saferound)
    14400    0.102    0.000    1.000    0.000 feature.py:38(__getitem__)
        2    0.002    0.001    0.874    0.437 __init__.py:24(<module>)
   179582    0.572    0.000    0.572    0.000 {round}
     3500    0.102    0.000    0.545    0.000 geometries.py:626(__getitem__)
    14400    0.196    0.000    0.505    0.000 field.py:20(__init__)
        1    0.001    0.001    0.477    0.477 pygmaps_ng.py:3(<module>)
        1    0.002    0.002    0.394    0.394 matplottools.py:3(<module>)
     6896    0.097    0.000    0.374    0.000 geometries.py:74(__init__)
    14400    0.105    0.000    0.372    0.000 feature.py:119(index)
187144/187135    0.352    0.000    0.355    0.000 {method 'join' of 'str' objects}
     2196    0.073    0.000    0.354    0.000 geometries.py:664(__getitem__)
    42598    0.187    0.000    0.313    0.000 base.py:25(_set_ptr)
        1    0.001    0.001    0.296    0.296 polysimplify.py:35(<module>)
     5697    0.117    0.000    0.292    0.000 geometries.py:268(_get_srs)
        1    0.002    0.002    0.282    0.282 pyplot.py:17(<module>)
        1    0.007    0.007    0.270    0.270 layer.py:199(get_geoms)
    14405    0.122    0.000    0.246    0.000 encoding.py:139(force_bytes)
     4800    0.014    0.000    0.228    0.000 field.py:94(value)
      263    0.003    0.000    0.220    0.001 re.py:230(_compile)
       20    0.015    0.001    0.217    0.011 __init__.py:1(<module>)
      125    0.002    0.000    0.216    0.002 sre_compile.py:567(compile)
     4800    0.051    0.000    0.214    0.000 field.py:56(as_string)
      121    0.001    0.000    0.214    0.002 re.py:192(compile)
       12    0.213    0.018    0.213    0.018 {open}
    15600    0.056    0.000    0.177    0.000 feature.py:25(__init__)
        1    0.002    0.002    0.174    0.174 colorbar.py:20(<module>)
    30159    0.128    0.000    0.167    0.000 errcheck.py:111(check_pointer)
        1    0.000    0.000    0.140    0.140 __init__.py:33(<module>)
      125    0.002    0.000    0.133    0.001 sre_parse.py:706(parse)
     4956    0.039    0.000    0.132    0.000 encoding.py:84(force_text)
        1    0.000    0.000    0.130    0.130 color_gen.py:1(<module>)
        1    0.000    0.000    0.130    0.130 __init__.py:6(<module>)
        1    0.002    0.002    0.129    0.129 brewer2mpl.py:1(<module>)
  355/125    0.005    0.000    0.128    0.001 sre_parse.py:317(_parse_sub)
     6896    0.044    0.000    0.127    0.000 geometries.py:239(geom_type)
  479/135    0.033    0.000    0.127    0.001 sre_parse.py:395(_parse)
        1    0.000    0.000    0.126    0.126 driver.py:2(<module>)
        1    0.003    0.003    0.125    0.125 ds.py:5(<module>)

Permafacture avatar Jun 17 '15 20:06 Permafacture