Segmentation failure on GFP-only glioblastoma cells with thin, complex morphology (confocal & widefield)

[nirosan10]

Describe the problem

Dear Cellpose team,

first of all, thank you for developing and continuously improving Cellpose and Omnipose — these tools are invaluable to the community.

I’m a PhD student working in neuro-oncology, and I’ve run into a persistent segmentation problem I haven’t been able to resolve using Cellpose, Omnipose, or custom-trained models. I’d like to share the details here in case someone has encountered a similar issue — and to kindly ask for guidance or ideas.

Biological context

• Cell type: Human glioblastoma (GBM) cells in co-culture with other neural cells (e.g. astrocytes)

• Label: Only cytoplasmic GFP (no DAPI, no membrane marker)

• Microscopy: Confocal and Widefield

• Data types: Single time point images, Live-cell time series (TrackMate)

• Image modality: 1-channel GFP (8-bit, 16-bit grayscale)

Morphological challenges

Cells exhibit highly variable morphologies, including:

•     Elongated somata
  

•     Very thin, branching processes (filopodia-/neurite-like)
  

•     Irregular borders
  

•     Heterogeneous signal intensity (both within and between cells)
  

  In co-culture, cells often overlap, or protrusions cross each other in 2D projections

What I’ve tried

**Pretrained models:**

•     All standard Cellpose and Omnipose models (cyto, nuclei, cyto2, style vectors)
  

  Custom training:

•     Multiple rounds of model training with >200 manually labeled GFP images
  

•     Used both Cellpose GUI and Python interface for training and inference
  

  Preprocessing:

•     Fiji-based contrast normalization, background subtraction, smoothing
  

•     Single z-slices vs. max projections
  

  Parameter tuning:

    **Exhaustive testing of:**
  
  
        diameter (fixed and auto)
  
        flow_threshold
  
        cellprob_threshold
  
    Tried with and without invert and style autodetection
  

  Data augmentation:

    During training: flipping, blurring, intensity jitter, rotations
  

Despite these efforts, the results remain inconsistent.

Observed problems

Oversegmentation: Cells with elongated processes are split into multiple masks, or protrusions are counted as        separate cells

Undersegmentation: In denser areas, adjacent cells merge into single masks

No stable configuration: Parameter tuning always improves one failure mode at the cost of the other

What I'm asking

I’d be very grateful for any input regarding:

Is this GFP-only, cytoplasmic signal type known to be problematic for Cellpose or Omnipose?

Would you recommend a specific strategy (e.g. using a 2-stage model, training on mask edges separately, etc.)?

Could the style classifier, style vector clustering, or deep feature encoding be used to better separate these morphologies?

Is there any advice on training set design or labeling strategy for thin, branching processes?

I’d be happy to share raw images, label sets, or training configs upon request — or even prepare a minimal reproducible example if helpful.

Thank you again for developing such a powerful and accessible tool — and thanks in advance for your time and consideration!

[mrariden]

• GFP problematic?: No, not necessarily.
• Strategy recommendation: Our training code using instance masks should be sufficient.
• Style vector/encoding: Not likely to be useful, no.
• Advice: Use a higher niter for images that contain cells with long processes. Also try using cp4 with finetuning, it sounds like you only have used cp3.

Unfortunately, segmenting long spindles that overlap is a hard task.

[mrariden]

closing due to inactivity