pyresample icon indicating copy to clipboard operation
pyresample copied to clipboard

Published 20 hours ago • verified publisher icon pytroll

Correct resampling of multiple swaths that do not fit into memory

Open cpaulik opened this issue 8 years ago • 6 comments

We are currently working on the problem of resampling multiple swaths in a correct manner. Our main concern are edge effects when stitching swaths or parts of orbits together. We mostly use a hamming window for resampling so we do not want to ignore the edges of our swaths. We do have a working solution for the problem and I am currently looking into the best way to open source it.

My main questions are:

  1. Would you consider this functionality a part of pyresample or should wrappers around pyresample manage inter-swath resampling.
  2. Are you aware of other projects that tackle the same issue which we could contact if your answer to question 1 is no.

Otherwise we will open source it as our own project and keep you informed if you are interested.

cpaulik avatar Jan 31 '17 13:01 cpaulik

@cpaulik Sorry for the late answer. I would say it would be a nice addition to pyresample! Do you have some examples ?

mraspaud avatar Apr 07 '17 07:04 mraspaud

I have the same use case of resampling swaths with high memory consumption.

I saw some dependencies on dask in pyresample. Would using dask/xarray be a solution to avoid loading everything in memory ?

@cpaulik I would be very interested in knowing more about your working solution.

floriankrb avatar Feb 26 '18 09:02 floriankrb

Yes, we're working with dask and xarray atm, and that helps with memory consumption. Feel free to join us on slack to discuss it, as it's still quite experimental (https://pytrollslackin.herokuapp.com/)

mraspaud avatar Feb 26 '18 09:02 mraspaud

@cpaulik I'm assigning you to this issue, feel free to contact us if you need help with a PR

mraspaud avatar Apr 26 '18 07:04 mraspaud

Thanks. Just don't expect a solution too soon. Since I first raised this issue two things have changed.

  1. I switched jobs and no longer have access to the code I talked about in the original post. This is a minor problem since we would have to rewrite most of it anyway for a successful integration in pyresample.
  2. I realized that we would have to extend pyresample in a way that the user can express how the points in the input datasets are related and how much overlap e.g. in time is necessary to perform a seamless resampling between parts of a swath.

Do you have any thoughts on 2.?

cpaulik avatar Apr 27 '18 11:04 cpaulik

No rush, we can wait :)

About 2. I'm wondering how that data structure would be implemented. An adjacency graph/matrix of sorts maybe ?

mraspaud avatar Apr 27 '18 11:04 mraspaud