tlambert03
Feature Request: Well Position Metadata
Description
This ticket is mostly an exploratory one to gather more information on well specific metadata for use in the bioio-nd2 Bioio plugin. Recently we have been adding metadata extraction to bioio-nd2 and one of the ideal metadata values is Row and Column. I was unable to find any reference to these values in the OME metadata or the raw metadata. The closest I got Was XYPosLoop (which is None for all of our internal files). I was hoping you might have some additional insight into how we might be able to parse these values?
it's a good question. We'll need to update this file of course, https://github.com/tlambert03/nd2/blob/main/src/nd2/_ome.py .... but I'm also not immediately sure where well position is stored in the raw metadata. I'm away this week so can't quickly collect a test dataset... do you happen to have a small file with a couple wells that I could look at?
Hey @tlambert03 apologies for the late response. The smallest multi well I was able to find is about 7gb. Is that viable for your usage? I can also provide just the metadata. The image in question is a multi-scene wellscan, however each position does not correlate to a specific well. (in some cases there are 2+ acquisitions of the same well). I do think we could probably derive well id using the physical position of acquisition along with the plate specifications but that seems like an unideal workaround and would require some assumptions.
Our scientists are planning to acquire a smaller multi-well Image for programatic testing though it may be a while...
depending on how it was acquired... it's conceivable that that might be the best way to do it. If it was acquired with JOBS, it's possible there might be some actual well info... but if was just a standard ND acquisition in elements, it's possible the positions really weren't much more than a regular grid in the standard XY stage position interface (which may or may not explicitly mention wells... i'm not sure).
there are a few lower level methods you could hit to dig through the outputs to see if you get anything interesting:
- ND2File.unstructured_metadata
- ND2File._rdr._cached_raw_metadata() or ND2File._rdr._cached_global_metadata
have a look in there and let me know if you think we might be able to extract something useful
I can confirm these images were acquired using JOBS. I previously looked into ND2File.unstructured_metadata and didn't see any specific reference. Ill take another look and get back to you!
do you have the .bin file associated with the job? I can probably actually parse that, the info might be in there rather than the nd2 file (though I know that would be a bit unfortunate/weird). we could at least confirm whether they share the same info, or if the job instruction .bin file contains stuff that doesn't make it to the acquired data
that bin file can actually be read with nd2, if you wanna try:
from nd2._parse._clx_lite import json_from_clx_lite_variant
p = Path('jobs_file.bin')
data = json_from_clx_lite_variant(p.read_bytes())
Here's some wellscan bin files we use. I found the summary particularly interesting
'Summary': '96 well plate, 1 point, 1 Lambda (Spinning Disk (3):740 LED), 1 Lambda (Spinning Disk (3):488-TL (660LP) 0.83x)'}}}
and
'Summary': '96 well plate, 40 wells, 24 sites (Covering), 1 Lambda (Spinning Disk (3):740 LED)'}}}
as well as some metadata here about WellplateDefinition
'WellplateDefinition': {'Key': 2,
'JobtaskKeyRef': 0,
'BlockNumber': 1,
'Order': 2,
'Version': 0,
'Name': 'Well_Plate',
'Data': [11,
Im not sure if we could correlate this directly with wells but we could probably use the summery to determine plate type and derive wells from the position indices
just an update here. I recently added a jobs() method that returns jobs data when present. That's where you'll find the wellplate info. Here's an example output from a nd2 file that uses jobs with a wellplate:
In [1]: with ND2File('tests/data/wellplate96_4_wells_with_jobs.nd2') as f:
...: print(f.jobs())
...:
{
'Job': {
'CustomDefinitions': {
'CustomWellplate': {
'CLxWellplate': {
'Uuid': 'd11550538ac34e6498222b8c7840b9e1',
'CompanyName': 'Custom wellplates',
'TypeName': '',
'ModelName': 'CellVis_96well',
'NamingMode': 2,
'XNamingType': 0,
'YNamingType': 1,
'CLxLayoutElement': {
'ActiveArea': {
'SLxShape': {
'Type': 1,
'Radius': 3105.0,
'CenterX': 0.0,
'CenterY': 0.0
}
}
},
'SLxShape': {
'Type': 2,
'RectL': -14300.0,
'RectT': -11360.0,
'RectR': 113300.0,
'RectB': 74390.0
},
'XWellOffset': 14300.0,
'YWellOffset': 11360.0,
'XCount': 12,
'YCount': 8,
'XDistance': 9000.0,
'YDistance': 9000.0
}
}
},
'Tasks': {
'WellplateDefinition': {
'Key': 1,
'JobtaskKeyRef': 0,
'BlockNumber': 1,
'Order': 1,
'Version': 0,
'Name': 'Plate',
'Data': {
'WellplateUUID': {
'UUID': 'd11550538ac34e6498222b8c7840b9e1'
}
},
'SlotConnections': [],
'Notes': '',
'NotesDetailed': 'Select a wellplate from the database or define
a custom wellplate. Then, define working area of a single well.',
'StateFlags': 0,
'LockableParams': {
'LockableParams': {'Show Plate Preview': False}
},
'Parameters': {'Parameters': {}}
},
'WellplateSelection': {
'Key': 2,
'JobtaskKeyRef': 0,
'BlockNumber': 1,
'Order': 2,
'Version': 0,
'Name': 'WellSelection',
'Data': {
'WellSelectionSettings': {
'Color': 4294901760,
'OrderingType': 0,
'SelectionMask': {
'Si': 96,
'St': 0,
'Co': 26,
'Sp': 3,
'Ma': [
9,
128,
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
}
}
},
'SlotConnections': {
'SlotConnections': {
'Slot0': {
'SlotType': 536873728,
'ParameterName': 'Plate.Wellplate',
'ParameterType': 0
}
}
},
'Notes': '',
'NotesDetailed': 'Specifies which wells of the selected
wellplate are actually used in the experiment.',
'StateFlags': 0,
'LockableParams': {
'LockableParams': {
'OrderingType': False,
'WellPlate': False,
'WellSelection': False
}
},
'Parameters': {'Parameters': {}}
},
'LoopWells': {
'Key': 3,
'JobtaskKeyRef': 0,
'BlockNumber': 1,
'Order': 3,
'Version': 0,
'Name': 'Wells1',
'Data': {
'CLxJobTask_LoopWells': {
'Storage': {'SplitStorage': False},
'AdvancedSettings': False,
'ForClass': '',
'ForLabel': ''
}
},
'SlotConnections': {
'SlotConnections': {
'Slot0': {
'SlotType': 536873216,
'ParameterName': 'WellSelection.Selection',
'ParameterType': 0
}
}
},
'Notes': '',
'NotesDetailed': 'Choose selection of wells to run the contained
tasks on. The number of wells can be further restricted by selecting specific
class (check the Advanced check box). Classes are defined with the Expression
task',
'StateFlags': 0,
'LockableParams': {
'LockableParams': {
'LabelList': False,
'WellLabelList': False,
'WellSelection': False
}
},
'Parameters': {'Parameters': {}}
},
'MoveToWellCenter': {
'Key': 4,
'JobtaskKeyRef': 3,
'BlockNumber': 1,
'Order': 1,
'Version': 0,
'Name': 'MoveToWellCenter',
'Data': {
'CLxJobTask_MoveToWellCenter': {
'MoveZ': 0,
'UsePFSOffset': 0
}
},
'SlotConnections': {
'SlotConnections': {
'Slot0': {
'SlotType': 536872960,
'ParameterName': 'Wells1.CurrentWell',
'ParameterType': 0
}
}
},
'Notes': '',
'NotesDetailed': 'Moves XY stage to the center of the current
well.',
'StateFlags': 0,
'LockableParams': {'LockableParams': {'SampleArea': False}},
'Parameters': {'Parameters': {}}
},
'CaptureCurrentOC': {
'Key': 5,
'JobtaskKeyRef': 3,
'BlockNumber': 1,
'Order': 2,
'Version': 0,
'Name': 'Capture',
'Data': {
'CLxJobTask_CaptureCurrentOC': {
'Storage': {
'StorageID': '',
'UseZSettings': 0,
'FillBlankMode': 1,
'SaveOptions': {'SaveOption': 0},
'Shading': {'ShadingMode': 0}
}
}
},
'SlotConnections': {'SlotConnections': {}},
'Notes': '',
'NotesDetailed': '',
'StateFlags': 0,
'LockableParams': {
'LockableParams': {
'JobTaskCaptureStorage': False,
'SaveOptions': False,
'ShadingOptions': False,
'Storage': False
}
},
'Parameters': {'Parameters': {}}
}
},
'PropertyDefinitions': {},
'Property': {
'Name': 'NewJob1',
'Desc': '',
'State': 0,
'ProgressProperties': 2883634,
'Usability': []
},
'ProgramParameters': {'Properties': [], 'Summary': ''}
},
'ProtectedJob': None,
'JobRunGUID': 'eb85f6f901a944d6b95b6b7b005d1851',
'ProgramDesc': {'JobDefType': 0}
}
that WellSelectionSettings.SelectionMask.Ma array is, by the way, a per-column bit mask of "row selected/not-selected"
So that should be where you go for this info. Even if I do eventually add some convenience on top of this data (to put that data into a more structured format) this API will remain, so for now, this is how you should check for well plates and extract all the info 👍
cheers