gplately icon indicating copy to clipboard operation
gplately copied to clipboard

Replace the guts of `gplately.download.DataServer` with `plate-model-manager`

Open brmather opened this issue 2 years ago • 8 comments

plate-model-manager is the new way (and the best way) to download plate models, age grids, and spreading rate grids. However, a lot of legacy workflows rely on the DataServer object in gplately. Currently, there are some advantages of retaining DataServer that perhaps will be propagated upstream:

  • pygplates.RotationModel is returned instead of a list of rotation filenames.
  • pygplates.FeatureCollection is returned instead of a list of topology filenames. Same applies for static polygons, coastlines, COBs, etc.
  • gplately.Raster is returned instead of a list of age grid / spreading rate grid filenames.
  • Missing features return an empty list and a DownloadWarning.

This wrapping of plate-model-manager implements lazy loading, where files are only downloaded when requested. These rotation models, topologies, etc. are accessed like attributes (using the @property decorator).

brmather avatar Nov 14 '23 04:11 brmather

IMPORTANT: I had to run gplately.download.clear_cache() to flush the cache of previously downloaded plate models, otherwise it would complain the folder (e.g. "Muller2019") already exists in the cache.

brmather avatar Nov 14 '23 04:11 brmather

@michaelchin is nearly ready to merge. The only problem I have is that gplately.download.clear_cache() needs to be run on the first run. Any idea how to do that or to get gplately.download.DataServer to overwrite the old plate models?

brmather avatar Aug 05 '24 08:08 brmather

clear_cache

Maybe replace _pooch.utils.make_local_storage(str(cache_path)) with Path(cache_path).mkdir(parents=True, exist_ok=True)???

michaelchin avatar Aug 05 '24 11:08 michaelchin

clear_cache

Maybe replace _pooch.utils.make_local_storage(str(cache_path)) with Path(cache_path).mkdir(parents=True, exist_ok=True)???

This is inside the clear_cache function, but is there a way to overwrite plate model files without clearing the cache?

brmather avatar Aug 05 '24 11:08 brmather

Hi @michaelchin - do you have any suggestions about the above issue? I think this could be a blocker for the next gplately release until we sort it out. Ideally it would be best if existing plate models downloaded by the old DataServer object could be overwritten...

brmather avatar Sep 11 '24 04:09 brmather

Hi @michaelchin - do you have any suggestions about the above issue? I think this could be a blocker for the next gplately release until we sort it out. Ideally it would be best if existing plate models downloaded by the old DataServer object could be overwritten...

Yes, if the new and old files have the same paths and names, I guess it would be possible to overwrite the old ones with the new ones. Otherwise, we may need to track the old files somehow...

michaelchin avatar Sep 11 '24 07:09 michaelchin

@michaelchin - Can you take over this PR? I don't have capacity any more and I am lost on how to get the new DataServer (powered by plate-model-manager) to overwrite old cached plate reconstructions.

brmather avatar Sep 12 '24 02:09 brmather

@michaelchin - Can you take over this PR? I don't have capacity any more and I am lost on how to get the new DataServer (powered by plate-model-manager) to overwrite old cached plate reconstructions.

Sure, no problem.

michaelchin avatar Sep 12 '24 05:09 michaelchin