Artificial Intelligence and Confocal Microscopy – What You Need to Know

Dynamic Signal Enhancement powered by Aivia boosts the denoising method with artificial intelligence (AI). Thus, Dynamic Signal Enhancement powered by Aivia even further strengthens the live-cell imaging capabilities of the STELLARIS platform requiring less trade-offs between cell viability, acquisition speed, image quality, and dimensionality.

DSE uses a rolling average calculation as its core algorithm. The ratio of a rolling window is that noise between adjacent imaging frames is random while biological signals are preserved. Accordingly, by averaging over several adjacent imaging frames with a rolling window, the relevant biological signal is accentuated while the unwanted noise is “averaged out”. In more expert terms, the DSE increases the signal-to-noise ratio (SNR) and, thus, leads to better image quality.

With Dynamic Signal Enhancement powered by Aivia, we now introduce a new option for the DSE which uses AI to interpolate frames between acquired raw data frames. Using the additional information from these  frames for rolling average calculation, DSE can now reach a higher SNR and better temporal dynamics at the same time.

Dynamic Signal Enhancement powered by Aivia is based on a specialized type of deep neural network architecture referred to as convolutional neural network (CNN). The very implementation of Dynamic Signal Enhancement powered by Aivia is based on the so-called “CAIN” network (Channel Attention Is All You Need for Video Frame Interpolation, Choi et al., 2020).

• What kind of movements?

Frame interpolation with Dynamic Signal Enhancement powered by Aivia works best for small, linear displacements of objects.

• What kind of specimens?

Dynamic Signal Enhancement powered by Aivia was trained on a large body of training data. We used more than 6,000 image triplets from more than 100 distinct microscopy time-lapses to fine-tune the underlying deep neural network on domain-specific data (ranging from cytoskeleton in cell culture to membrane staining in zebrafish retinae). Accordingly, we presume Dynamic Signal Enhancement powered by Aivia to generalize well on a wide range of specimen and applications.

• What kind of imaging data?

Dynamic Signal Enhancement powered by Aivia works on imaging data involving a time dimension, e.g., on xyt and xyczt scans. Furthermore, Dynamic Signal Enhancement powered by Aivia works for both FOV (Field of View) and resonant scanners, as well as for different modalities of the STELLARIS platform, e.g., confocal, DIVE and DLS. For the TauSense toolbox, Dynamic Signal Enhancement powered by Aivia works on all intensity data, i.e., on images for which life-time information was used for the acquisition but is not part of the resulting image (e.g., it works with TauSeparation, but not with TauContrast).

DSE (including Dynamic Signal Enhancement powered by Aivia) can be combined with LIGHTNING. This combination is especially rewarding when used in conjunction with the resonant scanner for fast and gentle live-cell imaging with multiple fluorophores. Furthermore, the processing of DSE and LIGHTNING can be chained, i.e., with a single “start” command the calculation of both image processing steps can be initiated.

Dynamic Signal Enhancement powered by Aivia has three different levels: “1”, “2”, and “3”. The Dynamic Signal Enhancement powered by Aivia level controls how many iterations of Dynamic Signal Enhancement powered by Aivia are applied on the image series, e.g., in case of Dynamic Signal Enhancement powered by Aivia level 1, a single iteration of Dynamic Signal Enhancement powered by Aivia is applied (and so on).

However, the Dynamic Signal Enhancement powered by Aivia level does not directly indicate the number of frames which get inserted between individual acquired raw data frames. In the single iteration for Dynamic Signal Enhancement powered by Aivia level 1, only a single image is inserted between individual acquired raw data frames. For Dynamic Signal Enhancement powered by Aivia level 2, in the first iteration a single interpolated frame is inserted between two adjacent raw data frames, and in the second iteration two additional interpolated frames are added (one between the “left” raw data frame and the interpolated frame from the first iteration, and the other one between the interpolated frame from the first iteration and the “right” raw data frame). So, in total there are three frames inserted with Dynamic Signal Enhancement powered by Aivia level 2. Finally, using the exact same reasoning from above, but with three iterations, Dynamic Signal Enhancement powered by Aivia level 3 adds a total of seven frames in between two adjacent raw data frames. In more general terms, the number of frames inserted between two adjacent raw data frames could be calculated by 2i-1 where the parameter i is the number of Dynamic Signal Enhancement powered by Aivia iterations.

Along those lines, it is frequently desired to also know the total number of frames used for a given rolling average window size and Dynamic Signal Enhancement powered by Aivia level. Thus, here is an example for using a DSE setting with five raw data frames and the resulting number of total frames used for rolling average calculation given different Dynamic Signal Enhancement powered by Aivia levels.

Table: Total number of frames (t) used for the DSE rolling window calculation as a function of the given DSE powered by Aivia levels (i) and raw frames (n); t = n + (2ⁱ – 1) * (n – 1).

The ratio of using higher Dynamic Signal Enhancement powered by Aivia levels is that the SNR can potentially be further improved by averaging over more Dynamic Signal Enhancement powered by Aivia-interpolated frames. This strategy is especially applicable when the size of the rolling window cannot be increased (i.e., using more raw images for the calculation is not feasible), because the best temporal dynamics is of key importance for the experiment.

No, Dynamic Signal Enhancement powered by Aivia does not increase the number of output frames, i.e., the number of output frames from the Dynamic Signal Enhancement (DSE) is unchanged – irrespective whether the Dynamic Signal Enhancement powered by Aivia option is activated or not. However, Dynamic Signal Enhancement powered by Aivia does transiently increase the number of frames, i.e., the number of frames used as input for calculating the rolling window of the DSE is increased. Since, however, the rolling window is moved in a way that the stride considers the Dynamic Signal Enhancement powered by Aivia level (i.e., it is proportional to how many frames were inserted between adjacent raw data frames), the number of output frames is independent of the selected Dynamic Signal Enhancement powered by Aivia level and thus invariant. In summary, Dynamic Signal Enhancement powered by Aivia helps to better preserve the temporal dynamics of the specimen by enabling to shrink the size of the rolling window (see also FAQ question 2), but without increasing the frame rate.

For the moment, Dynamic Signal Enhancement powered by Aivia can perform frame interpolation for the t-dimension. As Dynamic Signal Enhancement powered by Aivia only adds a single parameter to the Dynamic Signal Enhancement (number of t-interpolation iterations), Dynamic Signal Enhancement powered by Aivia is easy to use and can be activated by a single click.

Is it scientifically OK to denoise data with Dynamic Signal Enhancement powered by Aivia?

DSE uses a rolling average calculation as its core algorithm. The use of a rolling average is widely established as method for removing noise, with application fields ranging from imaging data to stock charts. The Dynamic Signal Enhancement powered by Aivia option adds a deep neural network to DSE processing. In contrast to many other deep learning algorithms for denoising, Dynamic Signal Enhancement powered by Aivia does not operate on single images and the spatial domain of the data, but rather on multiple input images and the temporal domain. Accordingly, Dynamic Signal Enhancement powered by Aivia excels on “temporal consistency” of the processed data. Furthermore, since Dynamic Signal Enhancement powered by Aivia is “only” adding some frames for the DSE rolling window calculation, there is always a blend of “ground-truth” raw-data frames and interpolated frames used for denoising. Finally, the user has always the option to compare DSE results with and without Dynamic Signal Enhancement powered by Aivia. Thus, especially for quantitative analysis (e.g., measuring fluorescence intensity levels), the user has easily the possibility to confirm the correct working of Dynamic Signal Enhancement powered by Aivia.

Can the Dynamic Signal Enhancement powered by Aivia algorithm be trained?

At the moment, we do not provide you with the opportunity to train the Dynamic Signal Enhancement powered by Aivia interpolation algorithm. However, we ship Dynamic Signal Enhancement powered by Aivia with a pre-trained deep learning model which first had been trained on an extensive publicly available video dataset and then fine-tuned via transfer learning on a domain-specific microscopy dataset. Accordingly, we presume Dynamic Signal Enhancement powered by Aivia to generalize well on a wide range of specimen and applications.

Does the weight parameter of Dynamic Signal Enhancement (DSE) have any impact on Dynamic Signal Enhancement powered by Aivia frame interpolation?

For DSE, not all frames are counted equally. There is a “weight” parameter used in DSE which determines how much frames close to the center of the rolling window, as well as how much more remote frames, contribute to this calculation.
Yet, the weight parameter is not considered when generating the Dynamic Signal Enhancement powered by Aivia frames via frame interpolation, because, in any case, only the two adjacent frames are used from the Dynamic Signal Enhancement powered by Aivia algorithm. When applying the rolling window calculation on both raw data frames and interpolated frames, the weight parameter is considered as described above for the rolling window calculation. In summary, the weight parameter is not used for the Dynamic Signal Enhancement powered by Aivia frame interpolation but used later during the rolling average calculation.

How does processing time scale with the number of Dynamic Signal Enhancement powered by Aivia iterations?

The time necessary for each Dynamic Signal Enhancement powered by Aivia round is not uniform. As every iteration practically doubles the number of input images for the next iteration, the following Dynamic Signal Enhancement powered by Aivia iteration takes twice as long as the previous round. Accordingly, while the second iteration would take twice the first interval, the third iteration would take quadruple the first interval. When combined with LIGHTNING, the processing time of both Dynamic Signal Enhancement powered by Aivia and LIGHTNING simply adds up, because they are executed sequentially. Moreover, since the interpolated frames are only transient (i.e., the number of output frames from the DSE is unchanged – irrespective of the application of the Dynamic Signal Enhancement powered by Aivia level), the processing time of LIGHTNING is independent from the employed Dynamic Signal Enhancement powered by Aivia level.