pyschism Generating GeoJSON to support NowCoast team

@Erin-nagel @agibbons-NOAA @josephzhang8 @cuill @feiye-vims @moghimis @gseroka @yangz888

Jul 19, 2022 | STOFS 3D GeoJSON Attendees: Adam Gibbons - NOAA Affiliate Erin Nagel - NOAA Affiliate Fei Ye John Kelley - NOAA Federal Linlin Cui Saeed Moghimi - NOAA Federal Y. Joseph Zhang Zizang Yang - NOAA Federal

Action items

Generating GeoJSON for STOFS3D

GeoJSON for Max disturbance
For max elev to add when that max value happened for specific
Include the start and end time of the forecast A set of hourly water surface elevation GeoJSON (2nd priority)

Metadata to GeoJSON

Valid time in GeoJson
Include rgb value + elevation value
Other metadata
When you click on a polygon what would you like to see?

Time stamp: hourly 2D outputs (STOFS-3D Atlantic) (2nd priority)
Adding current at 4.5m (3rd priority)

1st priority: Add above mention information to current max elev GeoJSON file

2nd: Then work on generating hourly GeoJSON files

Hi Saeed,

Thank you for reviewing the max water level STOFS3D web map service. Adam is currently working on the map interaction feature to query the service, however the GeoJSON only contains RGBA values rather than water level values and we would like to request some modifications to the current GeoJSON to include some additional attributes. Besides adding water level values is there any additional information your team would like to include?

Moving forward we would like to add an hourly STOFS3D service and I have included some general requirements and additional metadata for the GeoJSON files your team produces, please pass the requirements along to your team and let me know if you have any questions.

GeoJSON STOFS3D Product Requirements for nowCOAST:

Format: The filename must include the valid cycle time of the forecast for the first timestamp and the valid hourly forecast time or forecast time of the maximum water level disturbance for the second timestamp depending on the product Each feature-level object must include the following property names: "id","maxWaterLevel", "minWaterLevel", "verticalDatum", "units", "rgba" Example:

 properties": {
  "id": 1,
  "maxVal": 0.1,
  "minVal": 0.4,
  "verticalDatum": "NAVD88",
  "unit": "meters",
  "rgba": "#000080"
 }

Note: If there is only one water level elevation value per polygon, then include the property name "waterLevel" instead of "minWaterLevel" and "maxWaterLevel", but this attribute should be consistent for each product All property names must use the "camelCase" naming convention "id","maxWaterLevel", "minWaterLevel" properties must be encoded as true numeric values (i.e. not enclosed with quotes) and each feature-level object must have a unique id The top-level object should contain a coordinate reference system property Example:

"crs": { "type": "name", "properties": { "name": "EPSG:4326" } Polygons and MultiPolygons should follow the right-hand rule, as follows: The exterior ring should be counterclockwise. Interior rings should be clockwise. Example: The Shapely User Manual — Shapely 1.8.2 documentation All Polygons and MultiPolygons must be checked for validity and may not possess any overlapping exterior or interior rings, self intersections, duplicate nodes, sliver polygons, or gaps between polygons

Max Water Level Forecast Product

Suggested Product Path and Filename Format: s3://noaa-nos-stofs3d-pds/yyyymmdd/system_region_product_YYYYMMDDtHHz_YYYYMMDDtHHz.json

Example: s3://noaa-nos-stofs3d-pds/20220430/stofs3d_atlantic_maxwl_disturbance_ 20220430t12z_20220430t23z .json

Note: Filename must include the valid cycle time of the forecast for the first timestamp (e.g 20220430T12Z) and the valid forecast time of the maximum water level disturbance for the second timestamp (e.g 20220430t23z)

Hourly Water Level Forecast Product

Suggested Product Path and Filename Format: s3://noaa-nos-stofs3d-pds/yyyymmdd/forecast_cycle/region/system_region_fcst_YYYYMMDDtHHz_YYYYMMDDtHHz.json :

Examples:

s3://noaa-nos-stofs3d-pds/20220430/12/atlantic/stofs3d_atlantic_f001_20220430t12z_20220430t13z.json
s3://noaa-nos-stofs3d-pds/20220430/12/atlantic/stofs3d_atlantic_f002_20220430t12z_20220430t14z.json
s3://noaa-nos-stofs3d-pds/20220430/12/atlantic/stofs3d_atlantic_f003_20220430t12z_20220430t15z.json
s3://noaa-nos-stofs3d-pds/20220430/12/atlantic/stofs3d_atlantic_f004_20220430t12z_20220430t16z.json
s3://noaa-nos-stofs3d-pds/20220430/12/atlantic/stofs3d_atlantic_f005_20220430t12z_20220430t17z.json
s3://noaa-nos-stofs3d-pds/20220430/12/atlantic/stofs3d_atlantic_f006_20220430t12z_20220430t18z.json

Note: Filename must include the cycle time of the forecast for the first timestamp (e.g. 20220430t12z) and the valid hourly forecast time for the second timestamp (e.g 20220430t18z)

Thank you,

nowCOAST Team

Erin Nagel - NOAA Affiliate

Fri, Jul 8, 10:24 AM (11 days ago)

to me, John, Adam

Jul 19 '22 19:07 saeed-moghimi-noaa

@JohnGWKelley @erin-nagel @agibbons-NOAA @josephzhang8 @cuill @feiye-vims @moghimis @gseroka @yangz888

Nice website: https://www.gdacs.org/

Jul 20 '22 15:07 saeed-moghimi-noaa

Thanks, Saeed! - Zizang

On Wed, Jul 20, 2022 at 11:23 AM Saeed Moghimi @.***> wrote:

@JohnGWKelley https://github.com/JohnGWKelley @erin-nagel https://github.com/erin-nagel @agibbons-NOAA https://github.com/agibbons-NOAA @josephzhang8 https://github.com/josephzhang8 @cuill https://github.com/cuill @feiye-vims https://github.com/feiye-vims @moghimis https://github.com/moghimis @gseroka https://github.com/gseroka @yangz888 https://github.com/yangz888

Nice website: https://www.gdacs.org/

— Reply to this email directly, view it on GitHub https://github.com/schism-dev/pyschism/issues/44#issuecomment-1190424493, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TWSG2VI4MXY7BYFKRQOTVVAKW5ANCNFSM54BFEXFA . You are receiving this because you were mentioned.Message ID: @.***>

Jul 20 '22 15:07 yangz888

Thx Saeed; nice landing page.

-Joseph

Joseph Zhang Office: (804) 684 7466 Web: schism.wiki

From: Saeed Moghimi @.> Sent: Wednesday, July 20, 2022 11:23 AM To: schism-dev/pyschism @.> Cc: Y. Joseph Zhang @.>; Mention @.> Subject: Re: [schism-dev/pyschism] Generating GeoJSON to support NowCoast team (Issue #44)

[EXTERNAL to VIMS received message]

Nice website: https://www.gdacs.org/https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gdacs.org%2F&data=05%7C01%7Cyjzhang%40vims.edu%7C4dec1fdd782d45f9400e08da6a63cb3d%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C0%7C637939274084835039%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=urvQtObFBCALh15aU0IqCdUGNltkyuApP2dF3yJt%2BWg%3D&reserved=0

Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fschism-dev%2Fpyschism%2Fissues%2F44%23issuecomment-1190424493&data=05%7C01%7Cyjzhang%40vims.edu%7C4dec1fdd782d45f9400e08da6a63cb3d%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C0%7C637939274084835039%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DNeiPdZhln9tEUlgfWyEmGFmzBs0buipiOr5Jc1SUGU%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFBKNZ4HV3U7FVQE5F4KQOLVVAKW5ANCNFSM54BFEXFA&data=05%7C01%7Cyjzhang%40vims.edu%7C4dec1fdd782d45f9400e08da6a63cb3d%7C8cbcddd9588d4e3b9c1e2367dbdf1740%7C0%7C0%7C637939274084835039%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Whn0YCxZfVVdtXYX7sfxNf0yRpEvdaYGiGZaT5T8eks%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.@.>>

Jul 20 '22 16:07 josephzhang8

Hi @cuill

Any update on this issue?

Thanks, -Saeed

@feiye-vims @moghimis @gseroka @yangz888 @erin-nagel @agibbons-NOAA @josephzhang8

Jul 28 '22 20:07 saeed-moghimi-noaa

Hi Saeed,

The script to generate max disturbance was revised and added to the postprocessing workflow on Frontera. However, the STOFS3D forecast on Frontera was paused because of the texascale event this week.

I will let you know when the data is available on the S3 bucket next week.

Best, Linlin

Jul 28 '22 21:07 cuill

Hi @cuill

Thanks for the quick update.

Hi @yangz888 We may need to include this on the operational workflow as well.

Thanks, -Saeed

FYI @erin-nagel

Jul 28 '22 21:07 saeed-moghimi-noaa

Hi @cuill https://github.com/cuill & Saeed, Let's talk about it at the bi-weekly meeting next Tuesday. Thanks!

Zizang

On Thu, Jul 28, 2022 at 5:49 PM Saeed Moghimi @.***> wrote:

Hi @cuill https://github.com/cuill

Thanks for the quick update.

Hi @yangz888 https://github.com/yangz888 We may need to include this on the operational workflow as well.

Thanks, -Saeed

FYI @erin-nagel https://github.com/erin-nagel

— Reply to this email directly, view it on GitHub https://github.com/schism-dev/pyschism/issues/44#issuecomment-1198665863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TWSDGUMFGY3JXECPCBYLVWL54ZANCNFSM54BFEXFA . You are receiving this because you were mentioned.Message ID: @.***>

Jul 29 '22 12:07 yangz888

@yangz888

Sounds good! -Linlin

Jul 29 '22 13:07 cuill

Thanks to Linlin:

Update the script to add the TACC ssystem. However the TACC system is not working so there is no new GeoJSON
@cuill to share a sample GeoJSON with @erin-nagel

Rethink the disturbance to have more physical understandable (to public) definition: @josephzhang8

Maximum disturbance
- Colors over land mean water height above ground.
- Colors over water mean water height above NAVD88.

@yangz888

To include GeoJSON script to the WCOSS2 to have the files generated
@yangz888 will inform the team when the scripts implemented and the GeoJSON is being uploaded to S3 bucket.

Aug 01 '22 20:08 saeed-moghimi-noaa

GeoJSON files for max and hourly disturbance are available on S3 bucket:

https://noaa-nos-stofs3d-pds.s3.amazonaws.com/index.html#STOFS3D-Atl-VIMS/20220801/

stofs3d_atlantic_disturbance_20220731t00z_*.json stofs3d_atlantic_max_disturbance_20220731t01z_20220803t00z.json

Aug 02 '22 20:08 cuill

@erin-nagel @agibbons-NOAA @josephzhang8 @cuill @feiye-vims @moghimis @gseroka @yangz888

All,

First of all please make sure to communicate through this github issue for follow ups and updates:

Recent Email

STOFS3D Max Disturbance and Hourly GeoJSON Review
18 messages
________________________________________
Erin Nagel - NOAA Affiliate <[email protected]>	Tue, Aug 30, 2022 at 12:38 PM
To: Linlin Cui <[email protected]>
Cc: Saeed Moghimi - NOAA Federal <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Linlin,

Thank you for making the changes to STOFS3D GeoJSON content and adding the hourly forecast. After reviewing the max disturbance and hourly GeoJSON outputs and comparing it to the services provided by LSU displayed on the CERA website https://cera.coastalrisk.live/, it appears the GeoJSON is missing data, which is captured in the image below comparing the GeoJSON displayed in QGIS on the left and the LSU STOFS3D service on the right. 

So it may be possible that we are losing data during the mesh to contour conversion or possibly we need to use the data range from the model output, rather than a set data range of 0.5m to 2.9m, for the levels used to create the contours. 

The data range shown in the legend of LSU's STOFS3D service is < 0m to > 3m, do you know how the LSU team is creating their STOFS3D max disturbance service? Are they using the ADCIRC NetCDF output your team has generated to produce polygons? 

Let me know if you need any additional information and if you can look into this discrepancy.

Thank you,

Erin

-- 

Erin Nagel
Software Engineer, ERT
NOAA/NOS/OCS/CSDL

________________________________________
Linlin Cui <[email protected]>	Tue, Aug 30, 2022 at 1:16 PM
To: Erin Nagel - NOAA Affiliate <[email protected]>
Cc: Saeed Moghimi - NOAA Federal <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Erin,

So it may be possible that we are losing data during the mesh to contour conversion or possibly we need to use the data range from the model output, rather than a set data range of 0.5m to 2.9m, for the levels used to create the contours. 

Originally, we used Folium to generate maxelev html file. However, the html file was too large, then we specified the ranges (0.5-3.0) to show. I think this causes nodata areas. I can add model ranges back into GeoJSON file.

The data range shown in the legend of LSU's STOFS3D service is < 0m to > 3m, do you know how the LSU team is creating their STOFS3D max disturbance service? Are they using the ADCIRC NetCDF output your team has generated to produce polygons? 

LSU team are using ADCIRC NetCDF files we generated.

Best,
Linlin

From: Erin Nagel - NOAA Affiliate <[email protected]>
Date: Tuesday, August 30, 2022 at 12:39 PM
To: Linlin Cui <[email protected]>
Cc: Saeed Moghimi - NOAA Federal <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Subject: STOFS3D Max Disturbance and Hourly GeoJSON Review

[EXTERNAL to VIMS received message] 

Hi Linlin, 

Thank you for making the changes to STOFS3D GeoJSON content and adding the hourly forecast. After reviewing the max disturbance and hourly GeoJSON outputs and comparing it to the services provided by LSU displayed on the CERA website https://cera.coastalrisk.live/, it appears the GeoJSON is missing data, which is captured in the image below comparing the GeoJSON displayed in QGIS on the left and the LSU STOFS3D service on the right.  

[Quoted text hidden]

________________________________________
Saeed Moghimi - NOAA Federal <[email protected]>	Tue, Aug 30, 2022 at 2:31 PM
To: Linlin Cui <[email protected]>, Carola Kaiser <[email protected]>
Cc: Erin Nagel - NOAA Affiliate <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Linlin

I think CERA uses netcdf file to determine the range and does not use a set range. I cc here @Carola Kaiser  to chime in.

Best,
-Saeed

______________________________________________________
Saeed Moghimi, PhD
NOS Storm Surge Modeling Team Lead
Coastal Marine Modeling Branch, Coast Survey Development Laboratory, Office of Coast Survey at NOAA National Ocean Service.
Address: 1315 East West Hwy, Room 6607, Silver Spring, Maryland  20910 
Phone: ‪(302) 827-3986

The contents of this message are mine personally and do not necessarily reflect any position of NOAA.

[Quoted text hidden]

________________________________________
Carola Kaiser <[email protected]>	Wed, Aug 31, 2022 at 11:51 AM
To: Saeed Moghimi - NOAA Federal <[email protected]>, Linlin Cui <[email protected]>
Cc: Erin Nagel - NOAA Affiliate <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi All,

Yes, CERA uses the netcdf file to read the data. The range is not set to a fixed range with regard to the min/max values.

Carola

~~~~~~~~~~~~~~~~~~~~~~~~~~
Carola Kaiser, IT Consultant & GIS Web Development
Center for Computation & Technology
Louisiana State University
e-mail: [email protected]
CERA: https://cera.coastalrisk.live
[Quoted text hidden]

________________________________________
Erin Nagel - NOAA Affiliate <[email protected]>	Wed, Aug 31, 2022 at 1:11 PM
To: Carola Kaiser <[email protected]>
Cc: Saeed Moghimi - NOAA Federal <[email protected]>, Linlin Cui <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Carola,

Thanks for clarifying and how is your team using the ADCIRC grid netCDF file to display STOFS3D maximum disturbance and maximum water height in CERA?
Thank you,

Erin
[Quoted text hidden]

________________________________________
Saeed Moghimi - NOAA Federal <[email protected]>	Wed, Aug 31, 2022 at 2:09 PM
To: Carola Kaiser <[email protected]>
Cc: Linlin Cui <[email protected]>, Erin Nagel - NOAA Affiliate <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Linlin and Erin,

Perhaps we can do the same on the GeoJSON side. For instance, determine the range based on the data (mn and max over all) for each run cycle and keep it the same across all GeoJSONS for that specific run cycle. 

Since we have in the meta data it can be used to generate a new legend for each run cycle.

Does this make sense?

Thanks,
-Saeed 

______________________________________________________
Saeed Moghimi, PhD
NOS Storm Surge Modeling Team Lead
Coastal Marine Modeling Branch, Coast Survey Development Laboratory, Office of Coast Survey at NOAA National Ocean Service.
Address: 1315 East West Hwy, Room 6607, Silver Spring, Maryland  20910 
Phone: ‪(302) 827-3986

The contents of this message are mine personally and do not necessarily reflect any position of NOAA.

On Wed, Aug 31, 2022 at 11:51 AM Carola Kaiser <[email protected]> wrote:
[Quoted text hidden]

________________________________________
Linlin Cui <[email protected]>	Wed, Aug 31, 2022 at 5:25 PM
To: Saeed Moghimi - NOAA Federal <[email protected]>, Carola Kaiser <[email protected]>
Cc: Erin Nagel - NOAA Affiliate <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Saeed,

We may need to mask the data first to remove very large values on land boundary then calculate (min, max). I’ll take a look at the script. 

Best,
Linlin
[Quoted text hidden]

________________________________________
Carola Kaiser <[email protected]>	Thu, Sep 1, 2022 at 10:47 AM
To: Erin Nagel - NOAA Affiliate <[email protected]>
Cc: Saeed Moghimi - NOAA Federal <[email protected]>, Linlin Cui <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Erin, 
We are currently using a static classification with 32 levels. The levels have set intervals of 0.3 m. However, the minimum class contains all values < 0.3m and the maximum class all values > 10 ft. With that, we get map legend that stay the same for all runs, so that the runs can be easily compared to each other.

Regards, Carola
~~~~~~~~~~~~~~~~~~~~~~~~~~
Carola Kaiser, IT Consultant & GIS Web Development
Center for Computation & Technology
Louisiana State University
e-mail: [email protected]
CERA: https://cera.coastalrisk.live

From: Erin Nagel - NOAA Affiliate [mailto:[email protected]] 
Sent: Wednesday, August 31, 2022 12:12 PM
To: Carola Kaiser <[email protected]>
[Quoted text hidden]
[Quoted text hidden]

________________________________________
Erin Nagel - NOAA Affiliate <[email protected]>	Thu, Sep 1, 2022 at 11:41 AM
To: Carola Kaiser <[email protected]>
Cc: Saeed Moghimi - NOAA Federal <[email protected]>, Linlin Cui <[email protected]>, John Kelley - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Carola,

Thanks for the information and are you reading and displaying the data directly from the ADCIRC NetCDF or are you converting the ADCIRC NetCDF data into another format like ESRI Shapefile, to display max disturbance?

Erin 
[Quoted text hidden]

________________________________________
John Kelley - NOAA Federal <[email protected]>	Wed, Sep 7, 2022 at 3:52 PM
To: Linlin Cui <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>
Cc: Erin Nagel - NOAA Affiliate <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Linlin and Saeed,

Do you have an estimate on when we might have a new geoJSON file to test?

John

[Quoted text hidden]

________________________________________
Linlin Cui <[email protected]>	Wed, Sep 7, 2022 at 4:06 PM
To: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>
Cc: Erin Nagel - NOAA Affiliate <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi John,

Sorry for the delay in getting new geoJSON files. I have been tied up with other tasks. I will work on it the rest of the week. 

Before starting, I would like to know how many levels would your group like to be included in geoJSON files.  For reference, CERA has 32 levels. And let you know, increasing levels will increase file size accordingly.

Best,
Linlin
[Quoted text hidden]

________________________________________
John Kelley - NOAA Federal <[email protected]>	Wed, Sep 7, 2022 at 6:22 PM
To: Linlin Cui <[email protected]>
Cc: Saeed Moghimi - NOAA Federal <[email protected]>, Erin Nagel - NOAA Affiliate <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
LinLin,

Let's go with 32 levels as well.

We appreciate you working on this important task.

John

[Quoted text hidden]

________________________________________
Erin Nagel - NOAA Affiliate <[email protected]>	Wed, Sep 7, 2022 at 6:32 PM
To: Linlin Cui <[email protected]>
Cc: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Linlin,

Thanks for working on this, to handle the increased file size if possible can you use a different output file driver preferably a geopackage or shapefile.

Thank you,

Erin
[Quoted text hidden]

________________________________________
Linlin Cui <[email protected]>	Thu, Sep 8, 2022 at 1:54 PM
To: Erin Nagel - NOAA Affiliate <[email protected]>
Cc: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Erin,

Thanks for suggesting with a different output driver. I have generated three types (json, gpkg, and shp) of dataset for your test. You can download data here:

http://ccrm.vims.edu/yinglong/Cui/nowCOAST/

There are 32 polygons, with the first one covering data less than 0, and the last one covering greater than 3.0 m and less than model’s maximum. The interval is 0.1 m.

One method to further reduce the file size is to decrease decimals. But it seems to change the geometry type from “MultiPolygon” to “FeatureCollection” after doing make_valid. I am not sure if this will work on your end. 

Please let me know which data format you would prefer. Then I’ll add the revision to VIMS forecast workflow.
[Quoted text hidden]

________________________________________
Erin Nagel - NOAA Affiliate <[email protected]>	Fri, Sep 9, 2022 at 9:51 AM
To: Linlin Cui <[email protected]>
Cc: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Linlin,

Thank you for the quick turnaround and including all the requested outputs, we prefer a GeoPackage format for all outputs, and below is the visualized GeoPackage max disturbance output with a random color ramp applied, the output contains all of the data and attributes requested. 

However, the filename for the max disturbance output contains the start and end time of the forecast cycle (e.g stofs3d_atlantic_max_disturbance_20220807t01z_20220808t00z.gpkg) rather than the valid forecast time for when the maximum water level disturbance occurs within the forecast cycle. Let me know if it is possible to add that information to the filename, if this is not possible, I suggest only generating a water level GeoPackage file for each hour for the full forecast, instead of generating both the hourly water level and the max disturbance outputs.

Thank you,

Erin 

[Quoted text hidden]

________________________________________
Linlin Cui <[email protected]>	Fri, Sep 9, 2022 at 10:18 AM
To: Erin Nagel - NOAA Affiliate <[email protected]>
Cc: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Erin,

The time for max disturbance is spatially varying. We calculate max disturbance at each node based on its timeseries during forecast cycle. It is not possible to be included in the filename. 

Best,
Linlin

From: Erin Nagel - NOAA Affiliate <[email protected]>
Date: Friday, September 9, 2022 at 9:52 AM
To: Linlin Cui <[email protected]>
Cc: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Subject: Re: STOFS3D Max Disturbance and Hourly GeoJSON Review
Hi Linlin, 

Thank you for the quick turnaround and including all the requested outputs, we prefer a GeoPackage format for all outputs, and below is the visualized GeoPackage max disturbance output with a random color ramp applied, the output contains all of the data and attributes requested. 

However, the filename for the max disturbance output contains the start and end time of the forecast cycle (e.g stofs3d_atlantic_max_disturbance_20220807t01z_20220808t00z.gpkg) rather than the valid forecast time for when the maximum water level disturbance occurs within the forecast cycle. Let me know if it is possible to add that information to the filename, if this is not possible, I suggest only generating a water level GeoPackage file for each hour for the full forecast, instead of generating both the hourly water level and the max disturbance outputs.

Thank you,

Erin 
[Quoted text hidden]

________________________________________
Erin Nagel - NOAA Affiliate <[email protected]>	Fri, Sep 9, 2022 at 11:19 AM
To: Linlin Cui <[email protected]>
Cc: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Linlin,

Thanks for clarifying, let's move forward with the hourly water level geopackage files only. Let me know when the sample hourly files are available on http://ccrm.vims.edu/yinglong/Cui/nowCOAST/.

Thank you,

Erin
[Quoted text hidden]

________________________________________
Linlin Cui <[email protected]>	Fri, Sep 9, 2022 at 12:01 PM
To: Erin Nagel - NOAA Affiliate <[email protected]>
Cc: John Kelley - NOAA Federal <[email protected]>, Saeed Moghimi - NOAA Federal <[email protected]>, Adam Gibbons - NOAA Affiliate <[email protected]>
Hi Erin,

The sample files are ready on
http://ccrm.vims.edu/yinglong/Cui/nowCOAST/
[Quoted text hidden]

Sep 21 '22 13:09 saeed-moghimi-noaa

@erin-nagel @agibbons-NOAA @josephzhang8 @cuill @feiye-vims @moghimis @gseroka @yangz888

Hi Erin,

Please let @cuill and @yangz888 know about the status of your testing and if you need any changes regrading GeoJSON and GeoPackage. We are working closely with NOAA/NCO to finalize python requirement and we need to provide final python script to them asap.

Thanks, -Saeed

Sep 21 '22 13:09 saeed-moghimi-noaa

@cuill @yangz888 @agibbons-NOAA @josephzhang8 @cuill @feiye-vims @moghimis @gseroka @yangz888

STOFS3D Team,

For the max disturbance files would it be possible to add two additional attributes to the GeoPackage file for each polygon containing a minimum datetime and a maximum datetime, to capture the time range for when the spatially varying max disturbance occurs during the forecast cycle?

Otherwise we have no additional changes and please go ahead with posting the STOFS3D hourly disturbance GeoPackage files for the full forecast period (e.g. 72 hours) to the noaa-nos-stofs3d-pds S3 bucket.

Thank you, Erin

Sep 21 '22 14:09 erin-nagel

@cuill @yangz888 @agibbons-NOAA @josephzhang8 @cuill @feiye-vims @moghimis @gseroka @yangz888

Erin, Thanks for your prompt response.

I will let @cuill to comment on the datetime request.

Best, -Saeed

Sep 21 '22 14:09 saeed-moghimi-noaa

@erin-nagel @saeed-moghimi-noaa

I'll send the revised script for generating hourly GeoPackage files to Dan and Zizang. Meanwhile, I will think about how to add datetime information to the max disturbance file.

Thanks, Linlin

Sep 21 '22 14:09 cuill

@yangz888 Hi Zizang,

You can find the revised script here (on Frontera): /work2/07933/cuivims/ICOGS3D/Forecast/PostP/NewIO/gen_geojson.py

The command line to run it is the same as the old one. The only difference is it will generate gpkg files. Dan already tested it on Frontera. The runtime (~40 min on Frontera) is longer than the old one because more polygons are generated.

Let me know if you have any issues accessing the file or running it.

Thanks, Linlin

Sep 21 '22 18:09 cuill

Thanks @cuill for prompt response. Highly appreciated. -Saeed

Sep 21 '22 18:09 saeed-moghimi-noaa

Thanks, Linlin! I'll keep the progress posted! - Zizang

On Wed, Sep 21, 2022 at 2:18 PM cuill @.***> wrote:

@yangz888 https://github.com/yangz888 Hi Zizang,

You can find the revised script here (on Frontera): /work2/07933/cuivims/ICOGS3D/Forecast/PostP/NewIO/gen_geojson.py

The command line to run it is the same as the old one. The only difference is it will generate gpkg files. Dan already tested it on Frontera. The runtime (~40 min on Frontera) is longer than the old one because more polygons are generated.

Let me know if you have any issues accessing the file or running it.

Thanks, Linlin

— Reply to this email directly, view it on GitHub https://github.com/schism-dev/pyschism/issues/44#issuecomment-1254065674, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TWSEDDLP2LCQU4XLHSZ3V7NGP3ANCNFSM54BFEXFA . You are receiving this because you were mentioned.Message ID: @.***>

Sep 21 '22 18:09 yangz888

Dear Help Team,

FYI, I see the following on Acorn:

@.:/lfs/h1> pwd /lfs/h1 @.:/lfs/h1> ls -l total 0

Thanks, Zizang

Sep 21 '22 18:09 yangz888

Hi Linlin,

FYI that Acorn won’t be available until the end of this week. I’ll be able to test your new python code afterwards. I’ll keep you updated on the process.

Thanks, Zizang

---------- Forwarded message ---------From: Rick Beam - NOAA Affiliate < @.> Date: Wed, Sep 21, 2022 at 4:51 PM Subject: Acorn system still unavailable To: Davin Chan - NOAA Affiliate @.>

The Acorn system is still unavailable for job runs until Saturday 9/24/2022 We will send an email out when it is ready, If you attempt to run jobs that may fail or get deleted. We are preparing image updates and testing for new compilers etc...

On Wed, Sep 21, 2022 at 2:26 PM Saeed Moghimi @.***> wrote:

Thanks @cuill https://github.com/cuill for prompt response. Highly appreciated. -Saeed

— Reply to this email directly, view it on GitHub https://github.com/schism-dev/pyschism/issues/44#issuecomment-1254073110, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TWSFVVFSP3XF7NOU7R73V7NHLZANCNFSM54BFEXFA . You are receiving this because you were mentioned.Message ID: @.***>

Sep 21 '22 20:09 yangz888

@yangz888 Thanks for the updates.

Sep 21 '22 21:09 cuill

@erin-nagel @agibbons-NOAA @josephzhang8 @cuill @feiye-vims @moghimis @gseroka @yangz888

Linlin, Thanks a lot for your work on this topic. Highly appreciated.

Following conversation with Zizang, would you please place the latest geojson/package (please see the following on Frontera: /work2/07542/tknzyang/frontera/NPool/test_geoPkg/ dir_gpkg/*gpkg) somewher at the VIMS ftp site for Erin to test and provide feedback?

Thanks, -Saeed

Oct 07 '22 13:10 saeed-moghimi-noaa

Saeed,

GeoPackage files have been posted to noaa-nos-stofs3d-pds AWS S3 bucket since 9/24, I began testing nowCOAST@AWS operations for STOFS3D hourly product on 9/26, after making some adjustments on our end we were able to successfully process and display all 72 hour forecasts. We will let your team know when the service is available for review on one of our prototype map viewers, but here are two initial images during Hurricane Ian.

STOFS3D_sample1

STOFS3D_sample2

Oct 07 '22 14:10 erin-nagel

Hi @erin-nagel

@yangz888 was suggesting to do a final check before we pass the scripts to NCO to be included to the operational workflow.

@cuill Are there any changes to these files in comparison with the ones you passed to @erin-nagel before? If they are the same then we may not need to pass them to Erin for testing, right?

Thanks, -Saeed

Oct 07 '22 14:10 saeed-moghimi-noaa

@erin-nagel @saeed-moghimi-noaa @yangz888

The python script Zizang is using is the same as VIMS' version. So there are no changes to these files.

Maybe check if the file size is consistent with VIMS' version.

Oct 07 '22 14:10 cuill

Hi Erin,

Just to clarify, there are the following two sets of STOFS-3D-Atl run output on AWS S3: (1) https://noaa-nos-stofs3d-pds.s3.amazonaws.com/index.html#STOFS3D-Atl- VIMS/ The VIMS run, run cycle hour is t00z. (2) https://noaa-gestofs-pds.s3.amazonaws.com/index.html#_stofs-3D-Atlantic/ This directory contains outputs from the NOAA set up, run cycle hour is t12z; there are no gpkg files here yet.

I am running Linlin's updated gen_geopkg python script. I'll be able to place sample outputs on AWS S3 for you to test early next week. If they seem to be ok, we'll contact the NCEP NCO to start creating the gpkg files and make them available in (2).

Thanks, Zizang

On Fri, Oct 7, 2022 at 10:17 AM Erin Nagel @.***> wrote:

Saeed,

GeoPackage files have been posted to noaa-nos-stofs3d-pds AWS S3 bucket since 9/24, I began testing @.*** operations for STOFS3D hourly product on 9/26, after making some adjustments on our end we were able to successfully process and display all 72 hour forecasts. We will let your team know when the service is available for review on one of our prototype map viewers, but here are two initial images during Hurricane Ian.

[image: STOFS3D_sample1] https://user-images.githubusercontent.com/51652109/194574860-82147e58-a661-46ad-ab8a-e163c73cc1cc.JPG

[image: STOFS3D_sample2] https://user-images.githubusercontent.com/51652109/194574895-19baef76-fc06-4618-8c88-949c11289356.jpg

— Reply to this email directly, view it on GitHub https://github.com/schism-dev/pyschism/issues/44#issuecomment-1271656985, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TWSGAUI6OE6D2G5KQJCLWCAWGLANCNFSM54BFEXFA . You are receiving this because you were mentioned.Message ID: @.***>

Oct 07 '22 18:10 yangz888

Hi Zizang,

Thanks for clarifying, we currently only subscribe to the noaa-nos-stofs3d-pds bucket and process the VIMS run, so once you post the sample outputs to noaa-gestofs-pds, we can subscribe to that bucket and test the NOAA run, we will also deploy a map viewer next week so we can all review the hourly STOFS3D Atlantic web map service before you hand off the code to NCO.

Thanks everyone,

Erin

Oct 07 '22 18:10 erin-nagel

Sounds great! Thank you, Erin. - Zizang

On Fri, Oct 7, 2022 at 2:50 PM Erin Nagel @.***> wrote:

Hi Zizang,

Thanks for clarifying, we currently only subscribe to the noaa-nos-stofs3d-pds bucket and process the VIMS run, so once you post the sample outputs to noaa-gestofs-pds, we can subscribe to that bucket and test the NOAA run, we will also deploy a map viewer next week so we can all review the hourly STOFS3D Atlantic web map service before you hand off the code to NCO.

Thanks everyone,

Erin

— Reply to this email directly, view it on GitHub https://github.com/schism-dev/pyschism/issues/44#issuecomment-1271967791, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TWSBFX6KHRGIFNNJSGCTWCBWFHANCNFSM54BFEXFA . You are receiving this because you were mentioned.Message ID: @.***>

Oct 07 '22 18:10 yangz888

Hi Erin and John,

Please see the following sample input (netCDF) files and the created gpk files at the AWS S3: https://noaa-gestofs-pds.s3.amazonaws.com/index.html#_stofs-3D-Atlantic/stofs_3d_atl_sample_gpkg/, where source_data_to_create_pkg.nc.gz is the input file to Linlin's python script and the gpkg.gz files (total 72 files) are the outputs.

Please let me know if you have any questions.

Thanks, Zizang

On Fri, Oct 7, 2022 at 2:56 PM Zizang Yang - NOAA Federal < @.***> wrote:

Sounds great! Thank you, Erin. - Zizang

On Fri, Oct 7, 2022 at 2:50 PM Erin Nagel @.***> wrote:

Hi Zizang,

Thanks for clarifying, we currently only subscribe to the noaa-nos-stofs3d-pds bucket and process the VIMS run, so once you post the sample outputs to noaa-gestofs-pds, we can subscribe to that bucket and test the NOAA run, we will also deploy a map viewer next week so we can all review the hourly STOFS3D Atlantic web map service before you hand off the code to NCO.

Thanks everyone,

Erin

— Reply to this email directly, view it on GitHub https://github.com/schism-dev/pyschism/issues/44#issuecomment-1271967791, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TWSBFX6KHRGIFNNJSGCTWCBWFHANCNFSM54BFEXFA . You are receiving this because you were mentioned.Message ID: @.***>

Oct 11 '22 14:10 yangz888

All,

I have a STOFS3D demo available at https://137.75.94.147/gateway/, this link will only work on the internal NOAA network. There is an opacity slider in the menu dropdown, pop up information when you select the map, and service is time enabled.

Thank you, Erin

Oct 12 '22 18:10 erin-nagel

pyschism pyschism copied to clipboard

Generating GeoJSON to support NowCoast team

Action items

1st priority: Add above mention information to current max elev GeoJSON file

First of all please make sure to communicate through this github issue for follow ups and updates:

pyschism
pyschism copied to clipboard