Integration of Extra Oil with the NOAA database?

[ChrisBarker-NOAA]

now that you have oils in the adios_db JSON format:

https://github.com/OpenDrift/opendrift/tree/master/opendrift/models/openoil/adios/extra_oils

It would be nice to integrate that data with the "official" NOAA database:

https://adios.orr.noaa.gov

There isn't a lot to do for that, but:

Currently the oil records we got from the Norwegian Met office have IDs with a "NO" prefix.

We'll want to be careful about what's an updated record, and what's a new record.

You can comment on this gitHub repo:

https://github.com/NOAA-ORR-ERD/adios_oil_database

And this is a reminder to me to get the repo for the data behind that web site in a public repo -- stay tuned.

[gauteh]

Great, we'll make a PR to the data-repo once its public. We will probably use some other ID-naming scheme in the mean-time, so that once the new oils are included you can assign them new IDs.

[knutfrode]

Happ New Year, @ChrisBarker-NOAA !

Norwegian oils in the database have presently ID's from NO00001 to NO00127 Should we continue the numbering from there?

Are the IDs in the JSON-files the same as the ones used in the database, so that we have to make sure that they are all unique, and not conflicting with what you and other providers use? Or do you assign new ID's as they are added to the database?

We have also added the GENERIC oils from the old OilLibrary, e.g. https://github.com/OpenDrift/opendrift/blob/master/opendrift/models/openoil/adios/extra_oils/AD04000.json Are these ID's ok (AD04000 to AD04014) or should some other convention be used here (e.g. XX<number>)?

[ChrisBarker-NOAA]

Here is the data repo:

https://github.com/NOAA-ORR-ERD/noaa-oil-data

The ID in the data file should match the name of the file. And it is the ID we assign when putting it in our database.

the prefix e.g. "NOxxxxx" is a convenience for us so we can easily know where the data came from. So any records that you are updating should use the same prefix and ID, and any new ones should use NOxxxxx with a new unused number.

We don't have a good system in place for making sure that we don't reuse old IDs, but the prefix system makes it not too hard to at least manage your data :-)

For the generics, let's give them a "GN" prefix, so:

GN00001 GN00002 ...

We'll see how well a git merge does at updating!

Thanks!

[knutfrode]

Very good, then we are happy to be the official and sole provider of NOXXXXX-oils, and will make sure that IDs are not duplicated :-) And yes, GN sounds fine for GENERIC - we will commit these as well, as JSON-files. Btw, is there a particular reason why you did not already include these generic oils from the old OilLibrary? Are they not used with PyGNOME?

I cannot see the repo noaa-oil-data, perhaps it is still private to your organization?

[ChrisBarker-NOAA]

Thanks -- yes it was private -- they must have changed the default. You should be able to fix it now.

Btw, is there a particular reason why you did not already include these generic oils from the old OilLibrary? Are they not used with PyGNOME?

We do want a generic set for PyGNOME. But that set was kind of hacked together. We are actually embarking on a project right now to create a more robust set of Generic oils. e.g. carefully looking at data to make sure that those are actually representative.

If you are interested in, say, a "generic north sea" oil -- let us know, and we can work on developing that.

we are happy to be the official and sole provider of NOXXXXX-oils, and will make sure that IDs are not duplicated :-)

Thanks -- and yes, that's exactly why we've chosen to use the ID prefixes -- it's an easy way to give a "namespace" to different organizations.

Have you done all the oils that are in the NOFO reports here?

https://www.nofo.no/planverk/datasett/oljetyper-og-egenskaper/forvitringsstudier/

Also -- there is a lot of data in those reports that we did not capture in the old format, but we can use in the new -- have you added any of that new information? Let us know if you want help figuring out what to capture and how.

[knutfrode]

Great, I can see the repo now.

We will then soon commit the additional NO-oils. But then we should probably not commit old generic oils, but rather just keep them locally while waiting for the new generic oils. And the generic oils are the one we use most often, so it is good if they are consolidated.

I believe we have done all the listed NOFO-oils, and perhaps a couple more that we have received directly from NOFO.

We have not added any further parameters in addition to what was in the old CSV-files. But yes, it would be great with a little help here, so that we do it right.

E.g it could be very useful with a video meeting where we could walk through one of the PDF-reports (could look at it beforehand) and then together insert everything that can be added. After that, we should be able to repeat the same for all the other oils.

[ChrisBarker-NOAA]

thanks!

Yes, a meeting would be a good idea. I"ll check with the team here and we can plan it.

I'll reach out via email.

[knutfrode]

As a reminder: one question is whether maximum water uptake can be stored (and used)

[ChrisBarker-NOAA]

Absolutely!

Look at ADIOS online for a pretty complete oil, e.g.:

https://adios-stage.orr.noaa.gov/oils/EC04029

Look under Environmental Behavior / Emulsion Properties

And for the code that extract it:

adiso_db.computation.physicla_properties/.max_water_fraction_emulsion

We're actually working on that algorithm as we speak.