Semantic Segmentation Utilizing Foundation Models’ Features and Random Forest classifiers

Reconstructing organelles in three dimensions from electron microscopy requires segmenting every slice of an image stack. The images used in this project belong to the Ross Sea dinoflagellate, which has an exceptional ability to retain plastids and mitochondria that it has obtained from its haptophyte prey, Phaeocystis antarctica, for long periods. Within the dinoflagellate, the stolen organelles often undergo morphological changes; for instance, the mitochondria can become highly networked, making them challenging to segment accurately. Further, the dinoflagellate cells themselves are structurally complex and the stacks run to thousands of slices, making the dense manual annotation needed to train a supervised deep-learning model impractical. This project considers a different route. A napari plugin, developed in the project and released as FeatureForest, passes the internal image representations of pre-trained foundation models to a random forest classifier. A user draws a few strokes on a small number of slices, trains the classifier, inspects the result, and adds strokes wherever it fails. Once the result is good enough, the classifier segments the remaining slices unattended. The output is a segmentation of several organelle types at once and a 3D model of their arrangement in the cell.

The work was carried out as part of the first AI4Life open call, a Horizon Europe scheme that pairs life scientists holding an image-analysis problem with specialist analysis teams. A group at the Laboratoire Physiologie Cellulaire & Végétale (PCV), CNRS-Université Grenoble Alpes (Team Photosymbiosis) working on plankton cell biology and symbiosis, building a subcellular atlas of plankton symbioses, had a semantic segmentation problem which couldn’t be solved by available deep learning methods without having enough ground truth annotations. In particular, the complexity of the cells and size of the stacks render any complete manual annotation extremely time-consuming.

Image analysis and deep learning experts at Human Technopole took this project on. The work divided along disciplinary lines: the Grenoble group acquired the electron microscopy data, tested the plugin and advised on the analysis; the Human Technopole unit developed the method and the software. Imaging was performed on a nanocharacterisation platform in Grenoble, and the analysis ran on institutional GPU machines. The foundation models are used as published, without retraining; only the small classifier on top is trained, which keeps each training cycle short enough for a user to iterate interactively. The absence of ground truth was both the reason for the approach and the constraint that shaped its design.

From labels placed on eleven slices, the trained classifiers segmented three organelle classes across the full stack. They produced the 3D reconstruction of stolen plastids and mitochondria published in a Current Biology study of a kleptoplastidic dinoflagellate. The organelle volumes reported in that study were measured separately, by manual and semi-automatic segmentation. Accuracy varied by structure. Against manual segmentation of seven test slices, agreement was high for algal plastids and substantially lower for the two mitochondrial classes, so the approach replaced manual work for some organelles and supported it for others. The plugin is open source, installable from a public package index, documented with example notebooks, and archived with a DOI; the image stacks are deposited in a public electron microscopy archive under a no-rights-reserved licence. Both publications name the tool, state where it was used, and link the code. The plugin has since been reused in a later open-call project segmenting wood anatomy across 51 tree species, and in benchmarking it performed comparably to a deep-learning model trained on densely annotated data. The method avoids the compute of training a deep learning model but requires a GPU and substantial disk space for the extracted image representations.

Publications:

Seifi, Mehdi, Damian Dalle Nogare, Juan Manuel Battagliotti, Vera Galinova, Ananya Kedige Rao, Pierre-Henri Jouneau, Anwai Archit, Constantin Pape, Johan Decelle, and Florian Jug. “FeatureForest: the power of foundation models, the usability of random forests.” npj Imaging 3, no. 1 (2025): 32. https://doi.org/10.1038/s44303-025-00089-9,

Rao, Ananya Kedige, Daniel Yee, Fabien Chevalier, Charlotte LeKieffre, Marie Pavie, Marine Olivetta, Omaya Dudin et al. “Hijacking and integration of algal plastids and mitochondria in a polar planktonic host.” Current Biology 35, no. 11 (2025): 2509-2523. https://doi.org/10.1016/j.cub.2025.03.076)

Repository: https://github.com/juglab/featureforest

Image analysis use case: https://ai4life.eurobioimaging.eu/create-your-website-with-blocks/open-calls/use-cases/
Data: https://www.ebi.ac.uk/empiar/EMPIAR-12627/

Pipeline: https://zenodo.org/records/10830192

https://github.com/ai4life-opencalls/oc_1_project_52