VisCy

VisCy copied to clipboard

The goal of the analysis is to identify infected and uninfected cells in a FOV using either viral sensor fluorescence or label-free images with nuclear instance segmentation label to output a nuclear label identifying infected cells. The value of the nuclear label for each cell will indicate the following: 0: background 1: uninfected 2: Infected 3: unidentified (cases where the sensor has started relocalization, but the expression is not good enough to categorize as the infected state).

A manually annotated dataset with the above criteria is available here: /hpc/projects/intracellular_dashboard/viral-sensor/infection_classification/datasets/Exp_2023_09_28_DENV_A2_infMarked.zarr

The code is added to examples/infection_phenotyping folder on the branch.

• The file infection_annotator.py uses napari to help annotate the nuclear label.
• The file infection_classification_model.py is a work in progress on building a model for translation of instance segmentation of nuclear label into infection annotated label using the viral sensor fluorescence images or the label-free images.

@mattersoflight , @ziw-liu , presented here is a sample image of prediction logged during validation on tensorboard (this is a trial with manually input weights for cross-entropy loss calculation): The panel in each column shows:

1. sensor image,
2. 4-channel target annotation image
3. class 0 predicted image channel
4. class 1 predicted image channel
5. class 2 predicted image channel
6. class 3 predicted image channel The dataset used for training was chosen such that it has approximately the same number of infected and uninfected cells. Thus, the same cross entropy weights were given to mock the training. It can be seen from the image that class 2 is predicted better than class 1 as the sensor is localized in the nucleus and has high expression in class 2 cells. As a result, it looks easy to segment compared to uninfected cells. Will this be resolved if the cross entropy weights are best optimized or should we provide the nuclear segmentation as a second input channel with the sensor channel?

Updates from meeting with @mattersoflight and @ziw-liu today:

1. The expression of uninfected cells is very low, so perform feature enhancement to make sure the information from the fluorescence signal is properly used for training. Can either perform patch level normalization, or CLAHE can be used.
2. To detect the uninfected cell nucleus, either virtually stained nucleus, or nucleus distance map, or phase channel can be used as an extra input. It is recommended to use the phase channel to make this a hard problem to avoid the problem of overfitting. Also giving the segmented nucleus is not recommended as such training is sensitive to sharp edges.
3. Make sure to log the unet graphs on tensorboard.
4. Currently the batches are read such that one patch is taken from each image. Reprogram to use multiple patches from every image for training the model.

Thanks @mattersoflight . I have added the code I was preparing to run predictions on the data: example/infection_phenotyping/test_infection_classifier.py

Here is the path to the test dataset: /hpc/projects/intracellular_dashboard/viral-sensor/infection_classification/datasets/Exp_2024_02_13_DENV_3infMarked_test.zarr

This is the dataset used for training is here: /hpc/projects/intracellular_dashboard/viral-sensor/infection_classification/datasets/Exp_2024_02_13_DENV_3infMarked_trainVal.zarr

The checkpoint used for prediction: /hpc/projects/intracellular_dashboard/viral-sensor/infection_classification/models/sensorInf_phenotyping/logs_wPhase/version_34/checkpoints/epoch=99-step=300.ckpt

Rebase of this branch to main branch was required due to an issue encountered with the dataloader. While training the model, the first uninfected well was only utilized and not the second well with infected cells. The log from such training can be found here: /hpc/projects/intracellular_dashboard/viral-sensor/infection_classification/models/sensorInf_phenotyping/logs_wPhase/version_74. To resolve this the branch was rebased since the dataloader was working with the commits in the fcmae branch, which was merged into the main branch.

The branch is ready for merging into the main branch. Please let me know if any changes are pending. Can anyone review the branch so that we can merge it? Thanks!