Digital Evolution: How Computer Science and AI Are Driving Microscopy Innovation and Advancing Life Sciences

Biologists and other life scientists depend on microscopy to visualize cells and tissues in detail in biological samples. To characterize samples to understand biological processes, including disease mechanisms, researchers are increasingly employing a multiplex or multicolor microscopy technique. This allows you to stain different elements of the sample a different color – for example, the nucleus of a cell might be blue while the cell membrane is stained red.

Because each fluorophore has a characteristic emission spectrum – the range of wavelengths in which it emits light – the choice of fluorophores is critical when using different dyes at the same time to study interactions. As most fluorophores have a broad emission spectrum, it is important when using two or more fluorophores to select ones where the overlap of emission spectra is minimal. If there is overlap, their signals will interfere with each other in a phenomenon known as “crosstalk”, making the resulting data difficult to interpret.

It is therefore vital that fluorophores are well distinguished to achieve optimal results when multiplexing. Examples of when this is needed include the study of different immune cells in relation to key biomarkers in immuno-oncology studies, and the discrete visualization of multiple proteins to identify different types of neurons in complex synaptic networks in neuroscience studies (see Figure 1).¹

Figure 1: Adult rat brain visualized using multicolor or multiplex microscopy. The neuronal cells have been stained green with the fluorophore, Alexa Fluor488; the astrocytes have been stained red with a glial fibrillary acidic protein (GFAP) stain, and the cell nuclei have been stained blue with 4′,6-diamidino-2-phenylindole (DAPI). Image courtesy of Prof. En Xu, Institute of Neurosciences and Department of Neurology of the Second Affiliated Hospital of Guangzhou Medical University, China.¹

Algorithmic solutions providing new possibilities for interpretation of data

To help overcome the challenges of multiplex microscopy, automated tools have been developed that are helping researchers to algorithmically “unmix” spectral ranges – producing faster, smarter and higher-quality data and providing new possibilities for interpreting image data. For instance, Drs. Francesco Cutrale and Scott E. Fraser at the Translational Imaging Center of the University of Southern California (USC) have synthesized a new way of combining two existing methods and smartly automating it using simple algorithms to enable scientists to instantly acquire, store and analyze images produced using multiple fluorescent markers, in one go.^2,3

The first method involved is hyperspectral imaging, which was initially developed for remote sensing by airplanes flying over land or satellites flying around the globe. This kind of imaging incorporates the additional dimension of wavelength, in that it captures different wavelengths of light simultaneously in a large number of channels, rather than sequentially capturing single-color images respectively in a smaller number of channels.³

While this approach promised a better way of capturing multicolor images with high resolution, there were challenges in repurposing the technology for microscopy, not least of which was the light source, which in the original hyperspectral imaging method was the sun. The much lower light signal that came with microscopy meant having to address low signal to noise ratios in the fluorescence. Fluorescence hyperspectral imaging for microscopy was also impacted by speed and a limited photon budget.³

But these hurdles did not deter Professor Fraser, who managed to successfully implement hyperspectral detection for microscopes 20 years ago.³ He used complex mathematics to unmix the signals and separate out the contributions being made by each kind of spectral emission coming from the biological sample stained with multiple fluorophores. More recently, Dr. Cutrale has found a robust algorithm that is not only resistant to the types of noise encountered using hyperspectral microscopy but is also simple enough to describe not only the single pixel – which may be more susceptible to noise effects – but also the complete spectral composition of the entire sample.³

Removing the noise from hyperspectral microscopy

This is where the second method comes in. Like hyperspectral imaging, phasor analysis has also existed for decades, and is already well established for fluorescence lifetime imaging (FLIM). But it hadn’t been applied to hyperspectral microscopy, that is not until three years ago, when Dr. Cutrale started working on hyperspectral phasors.³ He discovered it was a powerful and effective tool for removing the noise from hyperspectral microscopy data.

But there was still a hurdle: researchers needed to learn about phasor analysis to an expert level in order to really understand how to manipulate the phasor to unmix the signals and denoise their data effectively. To overcome this, Dr. Cutrale integrated a hybrid linear unmixing algorithm and the partial automation of standard algorithms with the versatility and sensitivity of the phasor to unmix the signals and ended up with a solution that was faster, more sensitive, and much easier to use.

In fact, the speed, sensitivity, and utility gains of this combined method, with fluorescent hyperspectral microscopy and phasor analysis, allows researchers to capture and analyze their images in real-time, meaning that any misses can be rectified straightway by capturing new images of the same sample, something that would much harder if not impossible if the analysis had to be performed on the data much later – after the stains have lost their fluorescence.^2,3

Machine learning to map the subcellular distribution of human proteins

Rapidly advancing computational power is also helping to break new ground in the life sciences through advanced microscopy in the Human Protein Cell Atlas (HPA) program. Dr. Emma Lundberg, professor of cell biology proteomics at the KTH Royal Institute of Technology in Sweden and director of the HPA, is developing machine learning (ML) models to help map the subcellular distribution of most human proteins and study their movements and interaction in real-time.

The ML algorithms being developed by her team are helping to improve image segmentation in confocal microscopy and enable far more efficient data processing and analysis, including segmenting images into multiple sets of pixels,^4,5 recognizing patterns in protein distribution without bias and identifying even subtle changes in cellular morphology.⁴ Similarly, rare cell types can also be identified in these ML-based analyses.⁴

These ML models can also embed spatial information into a format that can be integrated with other types of molecular characterization, for instance, with proteomics data or single-cell sequencing data.⁴ Real-time analysis and frequent time-lapse imaging are also enabling the team to observe dynamic events in the cells, including rare events in cells that they can selectively image with AI-powered microscopy.⁴

Broadening access to microscopy by eliminating the complexity of multispectral imaging

Although hyperspectral unmixing has been used for some time in satellite images which used the sun as a light source, it proved challenging to develop this method for microscopy, which was hampered by a very low signal-to-noise ratio in fluorescence. In the end it took the combination of several methods, including phasor-based analysis and automated linear unmixing to arrive at a fast and reliable microscopy technique for hyperspectral unmixing in a “plug and play” format that a broader range of scientists can access.⁶ This is gentle on the sample because only a single image exposure is required.

Thanks to automated hybrid spectral unmixing, multicolor fluorescent imaging is now easier and faster, providing scientists an ideal solution for scanning large samples or capturing fast dynamic processes in live cells.⁶ Together with ML algorithms, these methods allow researchers to extract knowledge from their microscopy samples while they are still sitting at the microscope with their specimen, focusing on getting results instead of understanding their microscope, freeing them to collect better data and perform better science.

 References:

1.  Pelzer P. Multicolor microscopy: The importance of multiplexing. Leica Microsystems. https://www.leica-microsystems.com/science-lab/multicolor-microscopy-the-importance-of-multiplexing/. Published Jan 10, 2022. Accessed Jan 19, 2022.

2. Polakovic G. From detecting lung cancer to spotting counterfeit money, this new imaging technology could have countless uses. USC Stem Cell. https://stemcell.keck.usc.edu/from-detecting-lung-cancer-to-spotting-counterfeit-money-this-new-imaging-technology-could-have-countless-uses/. Published Feb 5, 2020. Accessed Feb 11, 2022.

3. Cutrale F, Trivedi V, Trinh L, et al. Hyperspectral phasor analysis enables multiplexed 5D in vivo imaging. Nat Methods . 2017. https://doi.org/10.1038/nmeth.4134  

4. Lundberg E, Leica Microsystems Corporate Communications. Applying AI and machine learning in microscopy and image analysis. Leica Microsystems. https://www.leica-microsystems.com/science-lab/applying-ai-and-machine-learning-in-microscopy-and-image-analysis/. Published Jan 10, 2022. Accessed Jan 21, 2022.

5. Petoukhov E. Using machine learning in microscopy image analysis. Leica Microsystems. https://www.leica-microsystems.com/science-lab/using-machine-learning-in-microscopy-image-analysis/. Published Jan 10, 2022. Accessed Jan 24, 2022.

6. Amon J, Laskey P. FluoSync – a fast and gentle method for unmixing multicolor widefield fluorescence images. White Paper. Leica Microsystems. https://go.leica-ms.com/FluoSync. Published Jan 10, 2022. Accessed Feb 11, 2022.