MouseLand
[BUG] Cellpose model fails on parts of image
Describe the bug I have trained a Cellpose 4 model on chunks (128*128 px) from brain images. When I apply the model to the full-size images, it generally works well. However, it fails completely on seemingly random parts of the images, which look rectangular (reminiscent of the chunk shapes).
I normalized my images before labelling, training and applying the model. The model was trained using recommended settings:
python -m cellpose --train --dir Z:\Labmembers\Ingvild\Cellpose\Aldh_model\training_chunks --test_dir Z:\Labmembers\Ingvild\Cellpose\Aldh_model\test_data --learning_rate 0.00001 --weight_decay 0.1 --n_epochs 100 --train_batch_size 1 --use_gpu --model_name 071825_aldh_cspam_100epoch --verbose --mask_filter _seg.npy
And applied to a whole image via the command line. Applying the model on the whole image in the GUI gives the same result.
I could not think of any good reason it should fail on the particular part of the image, as it looked fairly similar to other parts of the image. After training on several hundred chunks from different images, I also never had the model perform so poorly on any single chunk. I therefore cropped out the part of the image where this was happening and ran the model on only that crop, and in this case it works..
Screenshots
Brain image and resulting segmentation side-by-side, area where model underperforms circled:
Cropped view of the area ran in the Cellpose GUI with the same model:
I'm not sure if there's something obvious I am missing or if this is a bug, but would appreciate any help on this.
It's hard to say but could be related to the tiling steps as the image is passed into the net. There is a normalization that is done on the images at this step when you could try turning off since you're pre-normalizeing. It's --no_norm in the CLI
Thanks for the reply. When I try the --no_norm flag something even stranger happens. It runs normally and gives me "[WARNING] more than 65535 masks in image, masks returned as np.uint32", indicating to me that it is finding a lot of masks. However, the output tif and npy file has only a single mask. This only happens when I use the no_norm flag, not if I don't include it (everything else being exactly the same). Perhaps something happening during the outputing to 32bit?
Hi all,
Just reaching out as I am running into a very similar issue. I run out-of-the-box CPSAM on DAPI stained, large field-of-view slide scanner images (~3000x5000px, size = ~50MB), using all default inputs, both in GUI and CLI. Detection/segmentation works super well for most of the image, but appears to exclude seemingly random rectangular parts of the image (see screenshot below). If I run CPSAM on just those excluded parts, segmentation works well, as described above (@ingvildeb). The missing squares (or conversely the isolated ones that "work") appear to be ~220x220px, which I believe might correspond to a hard-coded tile size within CPSAM implementation (but I am writing entirely out of my depth here, so no idea if that is true or even remotely related).
Based on the above-mentionned advice (@mrariden), I've tried running 1/99th-percentile normalization separately and then running while turning off normalization, and I get the exact same behavior described above (@ingvildeb) : [WARNING] ... masks returned as np.uint32 etc.
So if anyone has any input/advice or workaround, that would be super helpful - thanks in advance!
@arthurgodino That's a very extreme example. Can you upload a sample that produces this kind of behavior? It will make the debugging a lot easier with this clear of an issue.
@mrariden thank you for your swift reply! Below is the image file for this example (as .png, can't upload the original .tiff as per github restrictions...). I observe the same behavior if running CPSAM on the .png image. This is DAPI staining of mouse ventral hippocampus btw - fairly standard brain tissue image and staining.
To reproduce the issue, the simplest way is through GUI: open GUI -> import image via drag-and-drop -> hit "run CPSAM". Also, I want to add that the segmented vs non-segmented "patches" are reproducible (the same regions/masks if I run the same image multiple times, through GUI or CLI, with .tiff or .png as input, etc). I'm running on MacOS (M2 chip) if that can be helpful - haven't tried on other platforms.
Thanks again very much for your help!
This appears to be very similar to an issue we are facing, though instead of failing to segment the cells it returns overlapping large cell poses that also seem to relate to tile size. This is with the default parameters, but no combination of threshold values eliminates this behavior. Let me know if this is distinctly different and I can make a new issue.
@codyslater Are you using any denoising or pre-processing on your image?
@arthurgodino Can you try running the model at a lower resolution? Either by downsampling the image or by using a higher diameter setting?
@mrariden no at the moment there is no additional processing happening to the image after the basic stitching of tiles at the microscope.
@mrariden I did some testing, and if I run the model on chunks of 128x128 pixels and reconstruct the image from the chunks, missing tiles do not occur. Example:
Zoomed in view of missing tiles:
As you can see, I also have something similar to the issue raised by @codyslater where we get some really large objects returned, in addition to the missing tiles. None of these issues are present with the tile-wise approach followed by reconstruction.
What we think is happening now is that the network is unsure about the size of objects. It makes sense that putting smaller tiles through the network would cue the network that the objects are smaller since there is less larger/long range information in the tiles.
We're still working on a comprehensive solution.
Thank you for reporting and looking into this.
I can confirm the behaviour for cellpose == 3.1.1.2 as well!
Input image of size ~4000x5000:
@m-albert does this still occur when you try different diameter sizes? Can you also post the cellprob and flows images for this problematic image?
Model: 'cyto'
Diameter 20:
Diameter 30:
Diameter 40:
The above images represent a nuclear label, here's the result using a cytoplasmic marker:
Okay this is weird, I'm observing this effect even on small images using cellpose version down to 2.3.2 (didn't test further down).
This is without using GPU on red hat linux.
Output of `pip list`
Package Version
----------------------------- --------------
affinder 0.4.0
aiohappyeyeballs 2.6.1
aiohttp 3.12.13
aiosignal 1.3.2
alabaster 1.0.0
anndata 0.11.4
annotated-types 0.7.0
antspyx 0.6.1
anyio 4.9.0
app-model 0.3.0
appdirs 1.4.4
argon2-cffi 25.1.0
argon2-cffi-bindings 21.2.0
array-api-compat 1.12.0
arrow 1.3.0
asciitree 0.3.3
asttokens 3.0.0
async-lru 2.0.5
attrs 25.3.0
babel 2.17.0
beautifulsoup4 4.13.4
bleach 6.2.0
bokeh 3.7.3
branca 0.8.1
Brotli 1.1.0
build 1.2.2.post1
cached-property 1.5.2
cachey 0.2.1
cellpose 2.3.2
certifi 2025.6.15
cffi 1.17.1
charset-normalizer 3.4.2
click 8.2.1
cloudpickle 3.1.1
colorama 0.4.6
colorcet 3.1.0
comm 0.2.2
contourpy 1.3.2
cycler 0.12.1
cytoolz 1.0.1
czifile 2019.7.2.1
dask 2024.10.0
dask-expr 1.1.16
dask-image 2024.5.3
dask-jobqueue 0.9.0
datashader 0.18.1
debugpy 1.8.14
decorator 5.2.1
defusedxml 0.7.1
Deprecated 1.2.18
distributed 2024.10.0
docstring_parser 0.16
docutils 0.21.2
exceptiongroup 1.3.0
executing 2.2.0
fasteners 0.19
fastjsonschema 2.21.1
fastremap 1.16.1
filelock 3.18.0
fill_voids 2.1.0
flexcache 0.3
flexparser 0.4
folium 0.20.0
fonttools 4.58.4
fqdn 1.5.1
freetype-py 2.5.1
frozenlist 1.6.0
fsspec 2025.5.1
geopandas 1.1.0
gmpy2 2.2.1
h11 0.16.0
h2 4.2.0
h5py 3.14.0
HeapDict 1.0.1
hpack 4.1.0
hsluv 5.0.4
httpcore 1.0.9
httpx 0.28.1
hyperframe 6.1.0
idna 3.10
imagecodecs 2025.3.30
imagecodecs-lite 2019.12.3
imageio 2.37.0
imagesize 1.4.1
importlib_metadata 8.7.0
importlib_resources 6.5.2
in-n-out 0.2.1
iniconfig 2.0.0
ipykernel 6.29.5
ipympl 0.9.7
ipython 9.3.0
ipython_pygments_lexers 1.1.1
ipywidgets 8.1.7
isoduration 20.11.0
itkwasm 1.0b189
itkwasm-downsample 1.7.0
itkwasm-downsample-wasi 1.7.0
jedi 0.19.2
Jinja2 3.1.6
joblib 1.5.1
json5 0.12.0
jsonpointer 3.0.0
jsonschema 4.24.0
jsonschema-specifications 2025.4.1
jupyter 1.1.1
jupyter_client 8.6.3
jupyter-console 6.6.3
jupyter_core 5.8.1
jupyter-events 0.12.0
jupyter-lsp 2.2.5
jupyter_server 2.16.0
jupyter_server_terminals 0.5.3
jupyterlab 4.4.3
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_widgets 3.0.15
kiwisolver 1.4.8
lazy_loader 0.4
legacy-api-wrap 1.4.1
llvmlite 0.44.0
locket 1.0.0
loguru 0.7.3
lz4 4.4.4
magicgui 0.10.1
MALDIpy 0.1.5
mapclassify 2.9.0
markdown-it-py 3.0.0
MarkupSafe 3.0.2
matplotlib 3.10.3
matplotlib-inline 0.1.7
matplotlib-scalebar 0.9.0
matplotlib-venn 1.1.2
mdurl 0.1.2
mistune 3.1.3
mpmath 1.3.0
msgpack 1.1.1
multidict 6.5.0
multipledispatch 0.6.0
multiscale_spatial_image 2.0.2
multiview-stitcher 0.1.29
munkres 1.1.4
napari 0.5.6
napari-console 0.1.3
napari-matplotlib 3.0.0
napari-ome-zarr 0.6.1
napari-plugin-engine 0.2.0
napari-plugin-manager 0.1.6
napari-spatialdata 0.5.4.post0
napari-svg 0.2.1
narwhals 1.43.1
natsort 8.4.0
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nest_asyncio 1.6.0
networkx 3.5
ngff-zarr 0.12.2
notebook 7.4.3
notebook_shim 0.2.4
npe2 0.7.8
numba 0.61.2
numcodecs 0.15.1
numpy 1.26.4
numpydoc 1.8.0
ome-zarr 0.10.2
opencv-python 4.11.0
opencv-python-headless 4.11.0
optree 0.16.0
overrides 7.7.0
packaging 25.0
pandas 2.3.0
pandocfilters 1.5.0
param 2.2.1
parso 0.8.4
partd 1.4.2
patsy 1.0.1
pexpect 4.9.0
pickleshare 0.7.5
pillow 11.2.1
PIMS 0.7
Pint 0.24.4
pip 25.1.1
pkgutil_resolve_name 1.3.10
platformdirs 4.3.8
pluggy 1.6.0
ply 3.11
pooch 1.8.2
prometheus_client 0.22.1
prompt_toolkit 3.0.51
propcache 0.3.1
psutil 7.0.0
psygnal 0.13.0
ptyprocess 0.7.0
pure_eval 0.2.3
pyarrow 20.0.0
pybind11 2.13.6
pybind11_global 2.13.6
pyconify 0.2.1
pycparser 2.22
pyct 0.5.0
pydantic 2.11.7
pydantic-compat 0.1.2
pydantic_core 2.33.2
Pygments 2.19.1
pyimzML 1.5.5
pynndescent 0.5.13
pyogrio 0.11.0
PyOpenGL 3.1.9
pyparsing 3.2.3
pyproj 3.7.1
pyproject_hooks 1.2.0
PyQt5 5.15.11
PyQt5_sip 12.17.0
pyqtgraph 0.13.7
PySide6 6.9.1
PySocks 1.7.1
pytest 8.4.1
python-dateutil 2.9.0.post0
python-dotenv 1.1.0
python-json-logger 2.0.7
pytz 2025.2
PyWavelets 1.8.0
PyYAML 6.0.2
pyzmq 27.0.0
qtconsole 5.6.1
QtPy 2.4.3
referencing 0.36.2
requests 2.32.4
rfc3339_validator 0.1.4
rfc3986-validator 0.1.1
rich 14.0.0
rich-argparse 1.7.1
roifile 2025.5.10
roman-numerals-py 3.1.0
rpds-py 0.25.1
scanpy 1.11.2
scikit-image 0.25.2
scikit-learn 1.7.0
scipy 1.15.2
seaborn 0.13.2
Send2Trash 1.8.3
session-info2 0.1.2
setuptools 80.9.0
shapely 2.1.1
shellingham 1.5.4
shiboken6 6.9.1
sip 6.10.0
six 1.17.0
slicerator 1.1.0
sniffio 1.3.1
snowballstemmer 3.0.1
sortedcontainers 2.4.0
soupsieve 2.7
spatial_image 1.1.0
spatialdata 0.4.0
spatialdata-plot 0.2.10
Sphinx 8.2.3
sphinxcontrib-applehelp 2.0.0
sphinxcontrib-devhelp 2.0.0
sphinxcontrib-htmlhelp 2.1.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 2.0.0
sphinxcontrib-serializinghtml 1.1.10
stack_data 0.6.3
statsmodels 0.14.4
superqt 0.7.5
sympy 1.14.0
tabulate 0.9.0
tblib 3.1.0
terminado 0.18.1
threadpoolctl 3.6.0
tifffile 2025.5.10
tinycss2 1.4.0
toml 0.10.2
tomli 2.2.1
tomli_w 1.2.0
toolz 1.0.0
torch 2.7.1
tornado 6.5.1
tqdm 4.67.1
traitlets 5.14.3
typer 0.16.0
typer-slim 0.16.0
types-python-dateutil 2.9.0.20250516
typing_extensions 4.14.0
typing-inspection 0.4.1
typing_utils 0.1.0
tzdata 2025.2
umap-learn 0.5.7
unicodedata2 16.0.0
uri-template 1.3.0
urllib3 2.5.0
vispy 0.14.3
wasmtime 33.0.0
wcwidth 0.2.13
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
wheel 0.45.1
wheezy.template 3.2.3
widgetsnbextension 4.0.14
wrapt 1.17.2
xarray 2024.11.0
xarray-dataclasses 1.9.1
xarray-datatree 0.0.14
xarray-schema 0.0.3
xarray-spatial 0.4.0
xyzservices 2025.4.0
yarl 1.20.1
zarr 2.18.7
zict 3.0.0
zipp 3.23.0
zstandard 0.23.0