ngff icon indicating copy to clipboard operation
ngff copied to clipboard

Published 20 hours ago • verified publisher icon ome

bioformats2raw.layout

Open joshmoore opened this issue 2 years ago • 21 comments

First update for supporting bioformats2raw output as described in #104. Introduces the concept of "transitional" metadata to allow readers to start supporting extensions that have grown in the community before they are added here.

Draft previwable under: http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/joshmoore/ngff/bf2raw/latest/index.bs#bf2raw

Screen Shot 2022-04-07 at 16 04 04

Note: currently builds on top of #110 to prevent conflicts.

joshmoore avatar Apr 07 '22 14:04 joshmoore

Pushed some schema fun, @will-moore, if you want to take a look. I need to force push away the commits from #110 but I'll hold off on that a bit.

joshmoore avatar Apr 21 '22 10:04 joshmoore

Pushed some schema fun, @will-moore, if you want to take a look. I need to force push away the commits from https://github.com/ome/ngff/pull/110 but I'll hold off on that a bit.

I think the latest commit is missing bf2raw.schema. Unbearable suspense !

sbesson avatar Apr 21 '22 11:04 sbesson

I think the latest commit is missing bf2raw.schema.

Thanks. Pushed.

Unbearable suspense ! :)

Compared to the debate?

joshmoore avatar Apr 21 '22 13:04 joshmoore

Now that #110 is merged, can you merge main into here so that the diffs are cleaner?

will-moore avatar Apr 27 '22 09:04 will-moore

can you merge main into here so that the diffs are cleaner?

Done, @will-moore

joshmoore avatar May 03 '22 11:05 joshmoore

Pushed a statement about the recent realization that we want this layout for previous versions as opposed to upcoming versions. There might be a function in bikeshed along the lines of "applicable-to-version" that we may want to make use of.

joshmoore avatar May 11 '22 10:05 joshmoore

  • Thanks for the comments, all. Pushed a few commits.
  • @sbesson: updated validation tests to match.

If we're starting to feel fairly confident of the change, I can start on the backport to 0.4 (and beyond?)

joshmoore avatar May 30 '22 10:05 joshmoore

Now I almost want to have https://github.com/ome/ngff/pull/128 in first to have a review link handy :)

sbesson avatar May 30 '22 10:05 sbesson

Review URL: http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/e2119e5ccd9c97da8717c53b889f377c39d121a9/latest/index.bs

joshmoore avatar May 30 '22 10:05 joshmoore

Automated Review URLs

github-actions[bot] avatar May 30 '22 10:05 github-actions[bot]

It seems the github-actions link there (http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/8c7b30c7dde15be0665fa34325391461d2fd02a2/latest/index.bs) doesn't show what's in this PR? E.g. tried searching that page for "Transitional"

will-moore avatar May 30 '22 11:05 will-moore

Testing to see if a re-open will re-generate the comment (after f5c1193ed011e28be8c2019fad5c1f810b78b640)

joshmoore avatar May 30 '22 13:05 joshmoore

One more time (from 3f10889bb30511b70b973ca6d483368e411d863a)

joshmoore avatar May 30 '22 13:05 joshmoore

Should be ok to give it a try, @will-moore.

joshmoore avatar May 30 '22 13:05 joshmoore

Are we expecting another github-actions comment to appear?

will-moore avatar May 30 '22 15:05 will-moore

No, the comments get updated in place. Moving forward, it should be the first comment on a PR directly under the description.

joshmoore avatar May 30 '22 15:05 joshmoore

LGTM 👍

will-moore avatar May 30 '22 15:05 will-moore

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/release-of-omero-5-6-5/68960/1

imagesc-bot avatar Jun 30 '22 14:06 imagesc-bot

Some of the latest 0.4 plates published at https://idr.github.io/ome-ngff-samples/ also include the bioformat2raw.layout e.g. https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0056B/7361.zarr

We have not produced any public examples of non-HCS Zarr datasets with this layout. As elicited in https://github.com/ome/ome-zarr-py/issues/210#issuecomment-1173841118 we have limited tooling that can write this layout. From the IDR front, extending https://github.com/ome/omero-cli-zarr/pull/118/ to also support the export of Fileset:<id> is the most promising way forward in my opinion. Otherwise, @pwalczysko has produced some examples derived from the public OME-TIFF samples during the testing of OMEZarrReader and we could publish elsewhere

sbesson avatar Jul 05 '22 10:07 sbesson

As we move towards a merging of this work, it might be worth also considering @melissalinkert's https://github.com/melissalinkert/ngff/commit/056b4e905a0edfaa275fd352888f5b7edd410adb (on top of this PR). It proactively adds a feature that's currently needed by clients of the bioformats2raw output to deal with the loading of large filesets (e.g. HCS) over latent connections (e.g. S3):

# in OME/.zattrs
{
  "series" : [ "0", "1", "..." ]
}

Options

Hold off

  • Pro: this PR captures the current state and therefore is useful.
  • Con: as currently written, the bump would require going to bioformats2raw.layout=4.

Cherry-pick here

  • Pro: Less churn with the spec
  • Pro: we could loosen the "MUST" on the integer-based naming of the series ("0", "1", ...)
  • Con: needs unification with bioformats2raw (or we loosen the MUST requirement on the series field)

Open questions

  • Should an error or warning be raised if the various representations of the fileset don't match?
    • the Zarr groups
    • the OME-XML
    • the new series

joshmoore avatar Jul 11 '22 18:07 joshmoore

Followed the cherry-pick strategy from above but updated the semantics to be more permissive of missing "series" indices. cc: @melissalinkert

joshmoore avatar Jul 18 '22 18:07 joshmoore

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ome-zarr-storage-structure-for-multiple-multiscale-images/71330/2

imagesc-bot avatar Sep 06 '22 13:09 imagesc-bot

Thanks for the feedback, @will-moore! Changes pushed.

joshmoore avatar Sep 15 '22 09:09 joshmoore

Likely a last call for comments here. I'm going to move on to the various PRs:

  • [x] https://github.com/ome/ZarrReader/pull/31
  • [x] https://github.com/glencoesoftware/bioformats2raw/pull/157
  • [x] https://github.com/ome/ome-ngff-validator/pull/13
  • [ ] https://github.com/ome/ome-zarr-py/pull/174
  • [ ] https://github.com/ome/napari-ome-zarr/pull/47
  • [ ] https://github.com/hms-dbmi/vizarr/issues/149
  • [ ] https://github.com/ome/napari-ome-zarr/issues/21

Note: https://github.com/ome/ome-zarr-metadata will not be used for this release.

joshmoore avatar Sep 15 '22 09:09 joshmoore

Is the proposal still to port this transitional metadata specification only to the current stable 0.4 specification?

Yes. I'll do once all edits are made (Hopefully this morning).

joshmoore avatar Sep 22 '22 06:09 joshmoore

From my POV, I think we are ready to move forward with this PR as well as the bioformats2raw PR itself. Python I think will follow relatively quickly and then we'll look into Javascript.

cc: @melissalinkert @sbesson

@tischi: do you think there is interest in trying to add this to MoBIE now? or wait for the non-transitional version?

joshmoore avatar Sep 22 '22 06:09 joshmoore

do you think there is interest in trying to add this to MoBIE now? or wait for the non-transitional version?

I would be interested adding this to the hackathon package in mobie-io.

tischi avatar Sep 22 '22 09:09 tischi

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/intermission-ome-ngff-0-4-1-bioformats2raw-0-5-0-et-al/72214/1

imagesc-bot avatar Sep 28 '22 18:09 imagesc-bot

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/using-bioformats2raw-for-creating-ome-zarr-scale-format-string/72716/6

imagesc-bot avatar Oct 13 '22 10:10 imagesc-bot