Ugf newgrid for ISatSS

[hopesea79]

I have modified the make_grid.py and imagery.py to incorporate the ugf-newgrid into ISatSS.

[deeplycloudy]

@hopesea79 Thanks for your contribution to help get this branch ready for use in ISatSS.

I think this summarizes your changes:

1. You propose to use the value of fill from the function argument instead of a hard-coded value. Good idea.
2. You want to be able to override the output filename, so that it is always the same, instead of varying as a function of time. I can see how that would be useful in ISatSS, but your change breaks the naming functionality expected by the public product feed at Unidata. Outside ISatSS, the filenames need to be specific about the product in a way that matches the convention of other public GOES imagery feeds. Can your change be made into an option that doesn't change the standalone-functionality?
3. You want to turn off scale and offset for the x and y coordinate variables, keeping them as float32 instead of as scaled int16. Is it possible to account for this in a change to ISatSS, instead?

[djhoese]

For point 2 @deeplycloudy, what about my ideas in #54. If someone can provide a pattern as their output filename then glmtools can format it with the available metadata including datetimes. I'm not sure if the filename stuff in this PR is the same as what I was asking about in #54, but thought I'd mention it for the flexibility it provides.

[deeplycloudy]

@djhoese thanks for calling my attention to that outstanding issue. I commented over there to indicate that I think it's a good idea. I think 2. above is an additional wrinkle, where we also need to also allow an output filename template in addition to the path template.

[djhoese]

output filename template in addition to the path template.

Or just one argument that represents the whole output path? Thanks for commenting on my issue. I'll see what I can do before leaving today.

[deeplycloudy]

I'm not picky about whether it's one argument or two … let's follow the wisdom of any available Unix conventions.

This also relates to #26 - how do we expose the GOES sector and platform ID (G16, G17) so that it can be used to generate the standard, ABI-like filenames if needed?

[dopplershift]

It's not elegant, but for my GOES restitching, I just have a function that handles making the name and there's a limited set of parameters I put into the template string:

https://github.com/Unidata/ldm-alchemy/blob/58ff661e7b3c16c83e2409012cb4b590a7b22a76/goes-restitch.py#L39-L75

[hopesea79]

hi, Eric,

Happy new year!

Sorry for the late reply.

Thank you so much for your reply.

Actually, all these changes I've made are for the interface of the ISatSS-glmtools system. Yes, you are right: they are maybe not so good for the standalone-functionality.

For 2. Here the "output_filename_prefix" is used to make the output filename vary as a function of time. For instance, "output_filename_prefix='GLM'+str(startstamp)". Then I modified the glmtools.make_grids to make sure that "output_filename_prefix" was used by the function "write_goes_imagery". Again, all these changes I've made are for the interface of the ISatSS-glmtools system. Yes, you are right: they are maybe not so good for the standalone-functionality.

For 3. In ISatSS, glmtools is called to do the gridding parts. The outputs from glmtools are only intermediate, which will be deleted soon once they are read by ISatSS. To save time, I turned off the scale and offset for all the fields. In ISatSS, the x and y coordinate variables can be set according to some configure files. That means no glmtools' x and y are directly needed by ISatSS. So the scale and offset of x and y can be turn on.

Thank you again.

Best regards. Feng Zhang e: [email protected] c: 980-999-1126

On Thu, Dec 19, 2019 at 5:13 PM Eric Bruning [email protected] wrote:

@hopesea79 https://github.com/hopesea79 Thanks for your contribution to help get this branch ready for use in ISatSS.

I think this summarizes your changes:

1. You propose to use the value of fill from the function argument instead of a hard-coded value. Good idea.
2. You want to be able to override the output filename, so that it is always the same, instead of varying as a function of time. I can see how that would be useful in ISatSS, but your change breaks the naming functionality expected by the public product feed at Unidata. Outside ISatSS, the filenames need to be specific about the product in a way that matches the convention of other public GOES imagery feeds. Can your change be made into an option that doesn't change the standalone-functionality?
3. You want to turn off scale and offset for the x and y coordinate variables, keeping them as float32 instead of as scaled int16. Is it possible to account for this in a change to ISatSS, instead?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deeplycloudy/glmtools/pull/59?email_source=notifications&email_token=AOCT7LHRYSY2B3SR425IKI3QZPWXZA5CNFSM4J5OIRVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHLG3FQ#issuecomment-567700886, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOCT7LGZ77AO6R2R3T2FK4TQZPWXZANCNFSM4J5OIRVA .

[deeplycloudy]

@hopesea79 I renamed this PR just now to reflect that it is a discussion about how to take the ugf-newgrid branch meant for standalone use and adapt it to ISatSS. I think anything ISatSS specific should stay on its own branch. However, some of the changes should go in ugf-newgrid directly for all users, as described below.

For 1., I would like to have these changes in ugf-newgrid, and would be happy to merge them as a standalone PR directly into ugf-newgrid.

For 2., let's revisit that once we see what @djhoese is working on re: #54. If you have input into that design, please comment on that thread. I think we should be able to accommodate @djhoese and your needs all at once. I think that change will also be merged directly to ugf-newgrid.

For 3., I think you're proposing to revert scale_and_offset to their original behavior, and not use the changes in this PR? That sounds good to me. Otherwise, if you want scale_and_offset=False you can pass it as part of the output_kwargs dict in the ISatSS call to grid_GLM_flashes. You can see how this is done on line 284 of examples/grid/make_GLM_grids.py. In either case, it would not require a change to ugf-newgrid, but rather a change to how ISatSS calls glmtools.

Let me know if this works for your needs, as I am very interested in making sure we have something that is stable and ready for ISatSS as soon as possible.

[deeplycloudy]

See #63 for changes to the default output path for #54. @hopesea79 This will impact ISatSS integration, but should be a very simple change. Instead of running make_GLM_grids.py with -o /my/output/path/ you will pass -o /my/output/path/{start_time:%Y/%b/%d}/{dataset_name} to keep the same behavior. The same template string should be passed to the outdir kwarg in glmtools.grid.make_grids.grid_GLM_flashes if you are calling glmtools directly.

[hopesea79]

Thank you for your informing. I appreciate it.

Best

Feng Zhang e: [email protected] c: 980-999-1126

On Sat, Feb 15, 2020 at 3:33 PM Eric Bruning [email protected] wrote:

See #63 https://github.com/deeplycloudy/glmtools/pull/63 for changes to the default output path for #54 https://github.com/deeplycloudy/glmtools/issues/54. This will impact ISatSS integration, but should be a very simple change. Instead of running make_GLM_grids.py with -o /my/output/path/ you will pass -o /my/output/path/{start_time:%Y/%b/%d}/{dataset_name} to keep the same behavior. The same template string should be passed to the outdir kwarg in glmtools.grid.make_grids.grid_GLM_flashes if you are calling glmtools directly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deeplycloudy/glmtools/pull/59?email_source=notifications&email_token=AOCT7LANLR4YSXI5MYTG6OTRDBGTHA5CNFSM4J5OIRVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL3WL6I#issuecomment-586638841, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOCT7LGQLC734PVUTQDKGGTRDBGTHANCNFSM4J5OIRVA .

[deeplycloudy]

I just merged ugf-newgrid into master, and created a new isatss-ugf-newgrid branch for use in making any remaining changes that are desired for integration into ISatSS. I suggest migrating these changes over to that new branch.

[hopesea79]

Hi, Eric,

Great! Thanks for the update.

-- Feng Zhang e: [email protected] c: 980-999-1126

On Fri, Feb 28, 2020 at 11:20 AM Eric Bruning [email protected] wrote:

I just merged ugf-newgrid into master, and created a new isatss-ugf-newgrid branch for use in making any remaining changes that are desired for integration into ISatSS. I suggest migrating these changes over to that new branch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deeplycloudy/glmtools/pull/59?email_source=notifications&email_token=AOCT7LFEFQUBZXRVLZLQ7VTRFE2SFA5CNFSM4J5OIRVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENJCMJQ#issuecomment-592586278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOCT7LFMPPAXHJLWO2CLILTRFE2SFANCNFSM4J5OIRVA .