flopy icon indicating copy to clipboard operation
flopy copied to clipboard

bug: very slow export with model_attributes_to_shapefile

Open RyanConway91 opened this issue 2 years ago • 1 comments

Describe the bug Not a true bug, just a performance FYI 'model_attributes_to_shapefile' gets hung up and did not write anything after ~20min for my large disv model. If I write my own version, its works pretty fast (<1 min)

f_out = r'C:\offline\mcwhpp\junk\mdl_shps\whpp.shp'
sim_ws = r'V:\23\62\1462\multi_comm_WHPP\mf6\calibration\current'
sim = fp.mf6.MFSimulation.load(sim_ws= sim_ws)
ml = sim.get_model('local')
grb = fp.mf6.utils.MfGrdFile(r'%s\mcwhpp_m.dis.grb'%sim_ws)
modelgrid = grb.modelgrid
epsg = 26915
modelgrid.set_coord_info(crs = pyproj.CRS.from_epsg(epsg))
if False:
    #never does anything
    fp.export.shapefile_utils.model_attributes_to_shapefile(path = f_out,ml = ml, modelgrid =modelgrid,package_names = ['npf'])
if True:
    #my version... works in ~1min
    dir_out = r'C:\offline\mcwhpp\junk\mdl_shps'
    d_pks = {'npf':{'k':ml.npf.k.array,'k33':ml.npf.k33.array}}
    for pk, d_att in d_pks.items():
        array_dict = {}
        for att,a in d_att.items():
            a_ls = a.reshape((modelgrid.nlay,modelgrid.ncpl))
            for l,_a in enumerate(a_ls):
                array_dict['%s_L%i'%(att,l+1)] = _a
        fp.export.shapefile_utils.write_grid_shapefile(r'%s\%s_grd.shp'%(dir_out,pk),mg = modelgrid, array_dict = array_dict)

Expected behavior I expect 'model_attributes_to_shapefile' to work more quickly

Desktop (please complete the following information):

  • OS: Windows 10
  • Version '3.5.0.dev0'

Additional context Add any other context about the problem here.

RyanConway91 avatar Aug 23 '23 16:08 RyanConway91

@RyanConway91, so far I have not been able to reproduce this. I used "test050_create_tests_circle_island" from our tests in autotest\regression\test_mf6.py, and increased the number of layers to 400, so that the model has over 2,000,000 cells.

Exporting with "model_attributes_to_shapefile" took 35 seconds. Exporting with the code you provided above took 17 seconds. Note that "model_attributes_to_shapefile" exports more data than your code does, exported shapefiles are 119 MB vs 82 MB ("model_attributes_to_shapefile" also exports icelltype). In this case "model_attributes_to_shapefile" is running a little slower than your code (3.4 MB/sec vs 4.8 MB/sec), though nowhere near what you are describing.

Can you provide an example project to help me reproduce the problem?

spaulins-usgs avatar Aug 29 '23 16:08 spaulins-usgs

Feel free to reopen if needed @RyanConway91

wpbonelli avatar May 23 '24 00:05 wpbonelli