Cellpose, StarDist, and the Deep Learning Cell Segmentation Revolution

Deep learning cell segmentation has turned one of biological imaging's most tedious tasks, tracing every cell or nucleus in a field of view by hand, into a largely automated step. Cellpose and StarDist are the two open-source tools most researchers reach for first, but each was built around a different assumption about cell shape, and picking the wrong one for a given image type can cost hours of avoidable rework.

The cell segmentation problem in biological imaging

Cell segmentation involves drawing an accurate boundary around every cell body, membrane, or nucleus in a microscopy image. This process is a prerequisite for nearly every quantitative measurement that follows: counting cells, measuring intensity, tracking morphology, or extracting features for downstream phenotypic analysis. Manual segmentation is accurate but does not scale past a handful of images, and classical automated methods such as thresholding and watershed algorithms struggle whenever cells touch, overlap, or vary in size and shape within the same image.

These classical approaches typically also require substantial manual parameter tuning for each new imaging modality, staining protocol, or cell type, which limits their usefulness for labs running varied experiments. A threshold tuned for one fluorescence channel or magnification often fails outright on a different channel, cell line, or acquisition setting, forcing an analyst to adjust parameters again for every new dataset rather than reusing a single validated pipeline. Deep learning cell segmentation approaches emerged specifically to remove this recurring parameter-tuning burden, capturing shape and boundary patterns directly from large, varied training datasets rather than relying on fixed rules about pixel intensity or geometry.

That generalization is exactly where Cellpose and StarDist made their names, and the shift toward algorithmic, data-driven analysis they represent sits within a broader set of biological imaging analysis tasks that AI has transformed over the past several years. Each tool approaches the underlying prediction problem differently enough that neither is a strict replacement for the other, which is why understanding both remains useful even for labs that ultimately settle on one as a default.

The stakes of getting segmentation right extend beyond the time saved on any single image. Every downstream measurement—cell counts, intensity ratios, morphology metrics, or feature vectors feeding a phenotypic classifier—inherits errors introduced in the segmentation step . Those errors compound silently if a lab has no routine way to check segmentation quality against ground truth. A generalist model that performs reliably across imaging sessions also makes results more comparable across a multi-site study or a longitudinal experiment, since analysts no longer need to readjust parameters that could otherwise introduce a hidden source of variability between batches.

How Cellpose and StarDist approach deep learning cell segmentation differently

Cellpose and StarDist are the two most established tools for deep learning cell segmentation, and their differing shape assumptions explain most of the practical differences researchers encounter between them.

Cellpose, developed at Janelia Research Campus, frames segmentation as a problem of predicting spatial flows that point toward each object's center. It then groups pixels that flow to the same point into a single cell instance. The developers trained the model on a dataset of more than 70,000 segmented objects spanning a wide range of cell types and imaging conditions, which enabled a single generalist model to precisely segment cells across image types without retraining or manual parameter adjustment. That same two-dimensional (2D) architecture was extended to three-dimensional (3D) volumes without requiring any 3D-labeled training data; the 2D model's learned representations were reused for volumetric segmentation tasks.

A follow-up release, Cellpose 2.0, added a human-in-the-loop training workflow that allows researchers to correct a small number of the model's predictions using their own images. Those corrections are used to fine-tune a custom model. The developers reported that this approach needed as few as 100–200 annotated regions to reach accuracy comparable to a fully trained specialist model. This makes it practical to adapt Cellpose to unusual cell types or staining protocols without a large annotation effort. Community-contributed training images have since been folded back into later Cellpose model releases, which is part of why the tool's generalist models have continued to broaden in scope since the original release.

StarDist takes a geometrically different approach, representing each nucleus as a star-convex polygon defined by a set of radial distances from a center point, rather than predicting a flow field. This shape representation, introduced by researchers working on star-convex object detection, was designed specifically to handle densely packed, roughly round nuclei. In these cases, pixel-grouping methods are prone to merging adjacent objects, while bounding-box detectors struggle with heavy overlap. A convolutional network densely predicts the polygon parameters for every pixel, and non-maximum suppression then resolves the final set of individual nucleus instances.

The method was later extended to 3D star-convex polyhedra for volumetric microscopy data, adapting to the anisotropic voxel spacing common in fluorescence z-stacks. StarDist has also been applied beyond fluorescence microscopy. Adaptations trained for histopathology imaging have performed competitively in nucleus segmentation and classification challenges on stained tissue sections, extending the same underlying shape assumption to a substantially different imaging modality than the one for which it was originally designed.

Comparing Cellpose and StarDist: Choosing the right segmentation tool

The practical choice between Cellpose and StarDist starts with the shape of the object being segmented (Table 1). Cellpose's flow-based representation makes no assumptions about convexity, so it handles irregular cytoplasmic shapes, elongated cells, and clustered cells that do not fit a simple polygon. Therefore, it is commonly used for whole-cell and cytoplasmic segmentation across a wide range of tissue and culture types. It also ships with separate pretrained models optimized for cytoplasm versus nuclei, allowing researchers to select the model that best matches the channel being segmented.

StarDist's star-convex representation is a better geometric fit for nuclei, specifically. This is particularly true in dense fields where nuclei touch or overlap heavily, as the polygon representation naturally resists merging adjacent objects, unlike pixel-based grouping. That advantage narrows for irregularly shaped or highly elongated nuclei, where the star-convex assumption itself becomes a limitation rather than a benefit, and for whole-cell or membrane segmentation tasks that StarDist was never designed to address.

Table 1: A qualitative comparison of Cellpose and StarDist across the criteria that typically drive tool selection.

Criterion	Cellpose	StarDist
Underlying shape representation	Flow field pointing to object centers	Star-convex polygon or polyhedron
Best suited to	Irregular, elongated, or clustered whole-cell shapes	Roundish, densely packed nuclei
Handling of touching objects	Generally strong across shape types	Particularly strong for convex nuclei
Dimensionality support	2D with a 3D extension	Native 2D and 3D models
Retraining requirement	Works out of the box; human-in-the-loop fine-tuning available	Works out of the box; supports custom training

Many imaging labs do not treat this as an either-or decision, as each method provides useful information (Figure 1). A common workflow uses StarDist for nucleus detection on a nuclear counterstain channel and Cellpose for whole-cell boundaries on a membrane or cytoplasmic marker in the same image. Both segmentation outputs are combined to assign cytoplasmic signals to the correct nuclei. Quantifying features from each segmented cell this way raises many of the same population-level analysis questions researchers already face when working with high-dimensional cytometry data, since both fields ultimately need to turn per-object measurements into biologically meaningful groups.

Figure 1: A side-by-side schematic comparing flow-based and star-convex polygon approaches to deep learning cell segmentation. Credit: AI-generated image created using Google Gemini (2026).

Practical considerations beyond raw segmentation accuracy also shape which tool a lab adopts first. Both tools represent mature, well-supported branches of deep learning cell segmentation, so the decision comes down to shape assumptions rather than one tool being more advanced than the other. Both are open source, run through accessible graphical interfaces and scripted application programming interfaces. Additionally, they both integrate with common bioimage analysis platforms, lowering the barrier for labs with limited computational infrastructure. Community size matters too: both tools have active user bases and are frequently updated, so troubleshooting an unusual failure case is rarely a dead end. Finally, either tool can typically be run on a single workstation graphics processing unit for datasets of a modest scale.

Newer approaches to deep learning cell segmentation: Cellpose 3 and foundation model adaptations

Cellpose 3, released by the original Cellpose developers, shifted focus from segmentation architecture toward the image quality problem that limits any segmentation model: noisy, blurry, or undersampled microscopy data. Rather than training a separate restoration model to reconstruct pixel values, the developers trained the tool to output images that a generalist segmentation model could segment well while remaining perceptually close to the original data. The result is packaged as one-click image-restoration buttons within the existing Cellpose interface. This matters practically because faster or lower-light acquisition, the kind that reduces phototoxicity in live-cell experiments, tends to produce exactly the noisy and undersampled images that earlier segmentation models struggled with most.

A separate line of newer tools adapts general-purpose vision foundation models to microscopy rather than building segmentation-specific architectures from scratch. Micro-SAM, for example, fine-tunes the Segment Anything Model (SAM), a foundation model originally trained on natural images, specifically for light and electron microscopy data. The developers found that default SAM already generalized remarkably well to microscopy despite its natural-image training set, and fine-tuning closed most of the remaining gap for cell and nucleus segmentation tasks. The tool implements both interactive segmentation, where a user supplies point or box prompts, and a fully automatic mode built around an additional decoder trained to directly predict object centers and boundaries.

This foundation-model approach points toward a future where a small number of broadly capable vision models, adapted for specific microscopy tasks, gradually replace the current landscape of multiple narrowly scoped segmentation tools. But, it supports the move without necessarily displacing purpose-built methods like Cellpose and StarDist in the near term. Together, these developments show that deep learning cell segmentation remains an active area of methodological research rather than a solved, static toolkit.

Adopting any of these tools productively follows roughly the same sequence regardless of which segmentation model a lab ultimately settles on:

1. Identify the specific object type and imaging modality—cell body or nucleus, 2D or volumetric—since that alone rules out one of the two core shape assumptions.
   
2. Run each tool's default pretrained model on a representative subset of images before committing to any custom training.
3. Compare the outputs against a small, manually annotated ground truth set rather than relying on visual inspection alone.
4. Apply human-in-the-loop or custom training only if the default model's errors are consistent enough that a small number of corrected images will meaningfully improve results.
5. Reassess whenever the imaging modality, staining protocol, or cell type changes meaningfully, since a model validated on one dataset does not automatically generalize to another.

Making deep learning cell segmentation work in a real imaging pipeline

Deep learning cell segmentation has removed the single biggest bottleneck in quantitative microscopy, but Cellpose and StarDist are not interchangeable defaults, and neither is a universal deep learning cell segmentation solution on its own. The right choice depends on whether the object being segmented is convex and roundish, like most nuclei, or irregular and variable, like most whole-cell boundaries, more than it depends on any single published accuracy figure.

The newest tools in this space, from Cellpose 3's restoration models to microscopy-tuned adaptations of general-purpose foundation models, extend rather than replace that choice. Understanding what each underlying representation assumes about cell shape remains the fastest way to pick correctly on the first attempt, a skill that fits into the broader computational toolkit now expected across life science research. Labs that build a habit of validating segmentation output against ground truth, regardless of which tool produced it, will get more reliable downstream results than labs that treat any single model as infallible.

This content includes text that has been created with the assistance of generative AI and has undergone editorial review before publishing. Technology Networks' AI policy can be found here.