Keywords: Parallel computing, Magnetic resonance imaging (MRI), Image reconstruction - analytical & iterative methods
Abstract: This work presents a parallel algorithm for implementing the nonuniform Fast Fourier transform (NUFFT) on Google's Tensor Processing Units (TPUs). TPU is a hardware accelerator originally designed for deep learning applications. NUFFT is considered as the main computation bottleneck in magnetic resonance (MR) image reconstruction when k-space data are sampled on a nonuniform grid. The computation of NUFFT consists of three operations: an apodization, an FFT, and an interpolation, all being formulated as tensor operations in order to fully utilize TPU's strength in matrix multiplications. The implementation is with TensorFlow. cm{Numerical examples show 20x sim 80x acceleration of NUFFT on a single-card TPU compared to CPU implementations. The strong scaling analysis shows a close-to-linear scaling of NUFFT on up to 64 TPU cores.} The proposed implementation of NUFFT on TPUs is promising in accelerating MR image reconstruction and achieving cm{practical runtime for clinical applications

Keywords: Shape analysis, Population analysis, Probabilistic and statistical models & methods
Abstract: This paper develops statistical tools for testing differences in shapes of chromosomes resulting from certain gene knockouts (KO), specifically RIF1 gene KO (RKO) and the cohesin subunit RAD21 gene KO (CKO). It utilizes a two-sample test for comparing shapes of KO chromosomes with wild type (WT) at two levels: (1) Coarse shape analysis, where one compares shapes of full or large parts of chromosomes, and (2) Fine shape analysis, where chromosomes are first segmented into (TAD-based) pieces and then the corresponding pieces are compared across populations. The shape comparisons – coarse and fine – are based on an elastic shape metric for comparing shapes of 3D curves. The experiments show that the KO populations, RKO and CKO, have statistically significant differences from WT at both coarse and fine levels. Furthermore, this framework highlights local regions where these differences are most prominent.

Keywords: Image segmentation, Microscopy - Light, Confocal, Fluorescence, Machine learning
Abstract: Recent microscopy imaging techniques allow to precisely analyze cell morphology in 3D image data. To process the vast amount of image data generated by current digitized imaging techniques, automated approaches are demanded more than ever. Segmentation approaches used for morphological analyses, however, are often prone to produce unnaturally shaped predictions, which in conclusion could lead to inaccurate experimental outcomes. In order to minimize further manual interaction, shape priors help to constrain the predictions to the set of natural variations. In this paper, we show how spherical harmonics can be used as an alternative way to inherently constrain the predictions of neural networks for the segmentation of cells in 3D microscopy image data. Benefits and limitations of the spherical harmonic representation are analyzed and final results are compared to other state-of-the-art approaches on two different data sets.

Keywords: Microscopy - Super-resolution, Image quality assessment, Other-method
Abstract: We propose the use of Flat Metric to assess the performance of reconstruction methods for single-molecule localization microscopy (SMLM) in scenarios where the ground-truth is available. Flat Metric is intimately related to the concept of optimal transport between measures of different mass, providing solid mathematical foundations for SMLM evaluation and integrating both localization and detection performance. In this paper, we provide the foundations of Flat Metric and validate this measure by applying it to controlled synthetic examples and to data from the SMLM 2016 Challenge.

Keywords: Single cell & molecule detection, Compressive sensing & sampling, Microscopy - Light, Confocal, Fluorescence
Abstract: Imaging transcriptomics (IT) techniques enable characterization of gene expression in cells in their native context by imaging barcoded mRNA probes with single molecule resolution. However, the need to acquire many rounds of high-magnification imaging data limits the throughput and impact of existing methods. We propose an algorithm for decoding lower magnification IT data than that used in standard experimental workflows. Our approach, Joint Sparse method for Imaging Transcriptomics (JSIT), incorporates codebook knowledge and sparsity assumptions into an optimization problem. Using simulated low-magnification data, we demonstrate that JSIT enables improved throughput and recovery performance over standard decoding methods.

Keywords: Machine learning, X-ray imaging
Abstract: Transfer learning with pre-trained neural networks is a common strategy for training classifiers in medical image analysis. Without proper channel selections, this often results in unnecessarily large models that hinder deployment and explainability. In this paper, we propose a novel approach to efficiently build small and well performing networks by introducing the channel-scaling layers. A channel-scaling layer is attached to each frozen convolutional layer, with the trainable scaling weights inferring the importance of the corresponding feature channels. Unlike the fine-tuning approaches, we maintain the weights of the original channels and large datasets are not required. By imposing L1 regularization and thresholding on the scaling weights, this framework iteratively removes unnecessary feature channels from a pre-trained model. Using an ImageNet pre-trained VGG16 model, we demonstrate the capabilities of the proposed framework on classifying opacity from chest X-ray images. The results show that we can reduce the number of parameters by 95% while delivering a superior performance.

Keywords: Probabilistic and statistical models & methods, Image segmentation
Abstract: Deep Neural Networks (DNNs) have become ubiquitous in medical image processing and analysis. Among them, U-Nets are very popular in various image segmentation tasks. Yet, little is known about how information flows through these networks and whether they are indeed properly designed for the tasks they are being proposed for. In this paper, we employ information-theoretic tools in order to gain insight into information flow through U-Nets. In particular, we show how mutual information between input/output and an intermediate layer can be a useful tool to understand information flow through various portions of a U-Net, assess its architectural efficiency, and even propose more efficient designs.

Keywords: Machine learning, Image segmentation, Ultrasound
Abstract: In this work, we propose accurate binary Depthwise Separable Convolutional Neural Networks (DSCNNs) for medical image segmentation. The networks are binarized by learning the distribution of weights and activations, and by using parameter-free skip connections in their encoder and decoder structure. We design full precision DSCNNs based on symmetric encoder-decoder, feature pyramid network with an asymmetric decoder, and spatial pyramid pooling with atrous convolutions strategies for image segmentation. The DSCNNs have 14X and 8X fewer number of model parameters and operations, respectively, than standard segmentation networks. The trained full precision DSCNNs are used as baselines to achieve accurate binary DSCNNs. The networks are trained on two medical ultrasound datasets, a public fetal skull dataset and a privileged bladder dataset. The accuracy of the binary DSCNNs are within a 3% drop from the full precision networks on both the medical datasets.

Keywords: Image segmentation, Machine learning, Brain
Abstract: Segmenting an entire 3D image often has high computational complexity and requires large memory consumption; by contrast, performing volumetric segmentation in a slice-by-slice manner is efficient but does not fully leverage the 3D data. To address this challenge, we propose a multi-dimensional attention network (MDA-Net) to efficiently integrate slice-wise, spatial, and channel-wise attention into a U-Net based network, which results in high segmentation accuracy with a low computational cost. We evaluate our model on the MICCAI iSeg and IBSR datasets, and the experimental results demonstrate consistent improvements over existing methods.

Keywords: Image segmentation, Ultrasound, Machine learning
Abstract: Ultrasound imaging is one of the most commonly used diagnostic tools to detect and classify abnormalities of the women breast. Automatic ultrasound image segmentation provides radiologists a second opinion to increase diagnosis accuracy. Deep neural networks have recently been employed to achieve better image segmentation results than conventional approaches. In this paper, we propose a novel deep learning architecture, a Multi-Scale Self-Attention Network (MSSA-Net), which can be trained on small datasets to explore relationships between pixels to achieve better segmentation accuracy. Our MSSA-Net integrates rich local features and global contextual information at different scales and applies self-attention to multi-scale feature maps. We evaluate the proposed MSSA-Net on three public breast ultrasound datasets and compare its performance with six state-of-the-art deep neural network-based approaches in terms of five metrics. MSSA-Net achieves best overall segmentation results and improves the second best approach by 1.21% for Jaccard Index (JI) and 0.94% for Dice’s Coefficient (DSC).

Keywords: Machine learning, Magnetic resonance imaging (MRI), Brain
Abstract: Accurately predicting the survival of patients with cancer has the potential to substantially enhance and customize the treatment strategies. Integrating and using all the patients' available data is essential to make the most accurate predictions. In this work, we gather clinical, imaging and genetic data into one mono-block multivariate survival analysis for patients with primary central nervous system lymphoma (PCNSL). As a first step, we select the best features from each pre-processed dataset. Then we assemble and use the resulting block to predict overall survival with a survival random forest algorithm. The assessment of the proposed method yielded a C-index of 0.776. We thus conclude that multimodal data integration significantly improve prediction performance.

Keywords: Brain, Magnetic resonance imaging (MRI)
Abstract: Increased age has arguably one of the largest known effects on brain structure in healthy adult populations, often affecting different brain regions at different rates, and showing numerous interactions. Modeling age appropriately can be challenging. Tensor-based morphometry produces voxelwise maps of regional inter-subject differences in the brain, often in relation to a single study-specific (3D) template. In studies with a wide age range, this single template may not be sufficient. Here, we create age-specific templates within smaller age bins (4D) to compare to the standard 3D model and evaluate the potential biases. We analyzed the morphological changes of nearly 26,000 subjects from the UK Biobank. We found that age-related biases that existed with a 3D template were minimized with the 4D template. For effective modeling of age across the lifespan, a single template appears suboptimal.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Brain
Abstract: Deep Learning for neuroimaging data is a promising but challenging direction. The high dimensionality of 3D MRI scans makes this endeavor compute and data-intensive. Most conventional 3D neuroimaging methods use 3D-CNN-based architectures with a large number of parameters and require more time and data to train. Recently, 2D-slice-based models have received increasing attention as they have fewer parameters and may require fewer samples to achieve comparable performance. In this paper, we propose a new architecture for BrainAGE prediction. The proposed architecture works by encoding each 2D slice in an MRI with a deep 2D-CNN model. Next, it combines the information from these 2D-slice encodings using set networks or permutation invariant layers. Experiments on the BrainAGE prediction problem, using the UK Biobank dataset, showed that the model with the permutation invariant layers trains faster and provides better predictions compared to other state-of-the-art approaches.

Keywords: Magnetic resonance imaging (MRI), Brain, Shape analysis
Abstract: We present a method to simultaneously learn several linear discriminative models with explicit information sharing. We use TV-L1-regularized Logistic Regression in conjunction with a Tikhonov regularization term expressing shared information across disorders. The weighting of the cross-disorder term is spatially adapted based on the local mutual information between linear models. We apply the model to mesh-based morphometric features from 14 subcortical structures in Parkinson’s and Alzheimer’s disease datasets, PPMI and ADNI. To assess the overall improvement in model performance with cross-disorder information sharing over the baseline TV-L1 model, we sample out-of-fold ROC AUC scores using a shuffle-split procedure. Beyond improved prediction, the procedure can be used to formally test for the presence of shared morphometric signatures across diseases in specific regions of interest. Significantly higher ROC AUC scores were found for Parkinson’s prediction accuracy when regularized with the Alzheimer’s model based on putamen and caudate morphometry.

Keywords: Brain, Classification, Machine learning
Abstract: Cortical folding, an essential characteristic of the brain cortex, shows variability across individuals. Plis de passages (PPs), namely annectant gyri buried inside the fold, can explain part of the variability. However, a systematic method of automatically detecting all PPs is still not available. In this paper, we present a method to detect the PPs on the cortex automatically. We first extract the geometry information of the localized areas on the cortex via surface profiling. Then, an ensemble support vector machine (SVM) is developed to identify the PPs. Experimental results show the effectiveness and robustness of our method.

Keywords: Computational Imaging, Microscopy - Light, Confocal, Fluorescence, Microscopy - Super-resolution
Abstract: Single-molecule localization microscopy (SMLM) is a fluorescence microscopy technique that achieves super-resolution imaging by sequentially activating and localizing random sparse subsets of fluorophores. Each activated fluorophore emits light that then scatters through the sample, thus acting as a source of illumination from inside the sample. Hence, the sequence of SMLM frames carries information on the distribution of the refractive index of the sample. In this proof-of-concept work, we explore the possibility of exploiting this information to recover the refractive index of the imaged sample, given the localized molecules. Our results with simulated data suggest that it is possible to exploit the phase information that underlies the SMLM data.

Keywords: Microscopy - Super-resolution, Cells & molecules, Machine learning
Abstract: Single molecule localization microscopy is an essential observation tool in biology that yields data in the form of point clouds. It is still limited by an anisotropic resolution and inhomogeneous labeling density. This issue can be addressed by reconstructing a single model from multiple aligned copies of the same particle. However, generic registration methods fail to align point clouds in the presence of anisotropic noise and outliers. Therefore, we propose an alignment method dedicated to a common type of particle geometry, namely cylindrical symmetry. We focus on the centriole, a fundamental macromolecular assembly with ninefold cylindrical symmetry. We design a neural network based on stacked PointNet architectures that estimates the center and axis of symmetry of individual particles in SMLM, in order to align them in the same canonical space. We demonstrate the robustness of our approach on simulated and real dSTORM data.

Keywords: Microscopy - Light, Confocal, Fluorescence, Cells & molecules, Image enhancement/restoration(noise and artifact reduction)
Abstract: Fluorescence microscopy is a key driving force behind advances in modern life sciences. However, due to constraints in image formation and acquisition, to obtain high signal-to-noise ratio (SNR) fluorescence images remains difficult. Strong noise negatively affects not only visual observation but also downstream analysis. To address this problem, we propose a blind global noise modeling denoiser (GNMD) that simulates image noise globally using a generative adversarial network (GAN). No prior information on noise properties is required. And no clean training targets need to be provided for noisy inputs. Instead, by simulating real image noise using a GAN, our method synthesizes paired noisy and clean images for training a denoising deep learning network. Experiments on real fluorescence microscopy images show that our method substantially outperforms competing state-of-the-art methods, especially in suppressing background noise. Denoising using our method also facilitates downstream image segmentation.

Keywords: Microscopy - Light, Confocal, Fluorescence, Image acquisition, Heart
Abstract: In vivo microscopy is an important tool to study developing organs such as the heart of the zebrafish embryo but is often limited by slow image frame acquisition speed. While collections of still images of the beating heart at arbitrary phases can be sorted to obtain a virtual heartbeat, the presence of identical heart configurations at two or more heartbeat phases can derail this approach. Here, we propose a dual illumination method to encode movement in alternate frames to disambiguate heartbeat phases in the still frames. We propose to alternately acquire images with a ramp and pulse illumination then sort all successive image pairs based on the ramp-illuminated data but use the pulse-illuminated images for display and analysis. We characterized our method on synthetic data, and show its applicability on experimental data and found that an exposure time of about 7% of the heartbeat or more is necessary to encode the movement reliably in a single heartbeat with a single redundant node. Our method opens the possibility to use sorting algorithms without prior information on the phase, even when the movement presents redundant frames.

Keywords: Microscopy - Light, Confocal, Fluorescence, Cells & molecules, Tracking (time series analysis)
Abstract: Tracking of subcellular structures displayed as spots in fluorescence microscopy images is important to quantify viral and cellular processes. We have developed a novel tracking approach for biological particles which uses deep learning for both particle detection and particle association. Our approach combines a domain adapted Deconvolution Network for particle detection with an LSTM-based recurrent neural network for tracking. Past and future information in both forward and backward direction is exploited by bidirectional LSTMs, and assignment probabilities are determined jointly across multiple detections. We evaluated the proposed approach using image sequences of the Particle Tracking Challenge as well as live cell fluorescence microscopy data of hepatitis C virus proteins. It turned out that our approach yields state-of-the-art results or improves the results compared to previous methods.

Keywords: Magnetic resonance imaging (MRI), Computer-aided detection and diagnosis (CAD), Prostate
Abstract: Multi-parametric MRI (mpMRI) is a powerful non-invasive tool for diagnosing prostate cancer (PCa) and is widely recommended to be performed before prostate biopsies. Prostate Imaging Reporting and Data System version (PI-RADS) is used to interpret mpMRI. However, when the pre-biopsy mpMRI is negative, PI-RADS 1 or 2, there exists no consensus on which patients should undergo prostate biopsies. Recently, radiomics has shown great abilities in quantitative imaging analysis with outstanding performance on computer-aid diagnosis tasks. We propose an integrative radiomics-based approach to predict the prostate biopsy results when pre-biopsy mpMRI is negative. Specifically, the proposed approach combines radiomics features and clinical features with machine learning to separate positive and negative biopsy groups among negative mpMRI patients. We retrospectively reviewed all clinical prostate MRIs and identified 330 negative mpMRI scans, followed by biopsy results. Our proposed model was trained and validated with 10-fold cross-validation and reached the negative predicted value (NPV) of 0.99, the sensitivity of 0.88, and the specificity of 0.63 in receiver operating characteristic (ROC) analysis. Compared with results from existing methods, ours achieves 11.2% higher NPV and 87.2% higher sensitivity with a cost of 23.2% less specificity.

Keywords: Computed tomography (CT), Lung, Computer-aided detection and diagnosis (CAD)
Abstract: Non-Small Cell Lung Cancer (NSCLC) is the world’s leading cause of cancer deaths. A significant portion of these patients develop recurrence despite curative resection. Prognostic modeling of recurrence free survival in NSCLC has been attempted using computed tomography (CT) imaging features. Radiomic features have also been used to identify mutation subtypes in various cancers, however, the implications of such features on eventual patient outcome are unclear. Studies have shown that genetic mutation subtypes in lung cancers (KRAS and EGFR) have imaging correlates that can be detected using radiomic features from CT scans. In this study, we provide a degree of interpretability to quantitative imaging features predictive of mutation status by demonstrating their association with recurrence free survival using a hierarchical CT radiomics - deep learning pipeline.

Keywords: Lung, Computed tomography (CT), Probabilistic and statistical models & methods
Abstract: Lung cancer is the leading cause of cancer death in Europe with an approximate 5-years survival rate of 13% from diagnosis. The potential of computational image analysis to provide decision support in oncology and the importance of identifying predictive and non-invasive biomarkers of disease progression and response to therapy has led to Radiomics. The objective of this study was to develop a delta-radiomics signature to predict survival and treatment response in patients with advanced non-small cell lung cancer (NSCLC) undergoing immunotherapy. Pre-treatment and first follow-up CT images and intra-nodular and peri-nodular regions from 88 patients have been used to calculate delta-features. The delta-radiomics signature significantly stratified high- and low-risk patients (p = 0.018), it was significantly associated with Overall Survival (p = 0.03) and it predicted responders with an area under the receiver operating characteristic curve (ROC-AUC) of 0.76 in an independent test set. The results demonstrate the potential of delta-radiomics to be an early biomarker of immunotherapy response.

Keywords: Machine learning, Computed tomography (CT), Image segmentation
Abstract: Genetic profiling and characterization of lung cancers haverecently emerged as a new technique for targeted therapeu-tic treatment based on immunotherapy or molecular drugs.However, the most effective way to discover specific genemutations through tissue biopsy has several limitations, frominvasiveness to being a risky procedure. Recently, quanti-tative assessment of visual features from CT data has beendemonstrated to be a valid alternative to biopsy for the diag-nosis of gene-addicted tumors. In this paper, we present adeep model for automated lesion segmentation and classifi-cation as gene-addicted or not. The segmentation approachextends the 2D Tiramisu architecture for 3D segmentationthrough dense blocks and squeeze-and-excitation layers,while a multi-scale 3D CNN is used for lesion classifica-tion. We also train our model with adversarial samples, andshow that this approach acts as a gradient regularizer andenhances model interpretability. We also built a dataset, thefirst of its nature, consisting of 73 CT scans annotated withthe presence of a specific genomics profile. We test our ap-proach on this dataset achieving a segmentation accuracy of93.11% (Dice score) and a classification accuracy in identify-ing oncogene-addicted lung tumors of 82.00%.

Keywords: Machine learning, Computed tomography (CT), Image-guided treatment
Abstract: Accurate pre-operative overall survival (OS) prediction of gastric patients is of great significance for personalized treatment. To facilitate improvement of survival prediction, we propose a novel joint multi-task network equipped with multi-level features simultaneously predicting clinical tumor and node stages. Two independent datasets including a training set (377 patients) and a test set (122 patients) are used to evaluate our proposed network. The results indicated that the multi-task network exploits its recipe by capturing multi-level features, and sharing prognostic information from correlated tasks of clinical stages prediction, which enable our network to predict OS accurately. Our method outperforms the existing methods with the highest c-index (training: 0.73; test: 0.72). Meanwhile, our method shows better prognostic value with the highest hazard ratio (training: 3.77; test: 4.28) for dividing patients into high- and low-risk groups.

Keywords: Image registration, Machine learning, Magnetic resonance imaging (MRI)
Abstract: We introduce a strategy for learning image registration without acquired imaging data, producing powerful networks agnostic to magnetic resonance imaging (MRI) contrast. While classical methods accurately estimate the spatial correspondence between images, they solve an optimization problem for every new image pair. Learning methods are fast at test time but limited to images with contrasts and geometric content similar to those seen during training. We propose to remove this dependency using a generative strategy that exposes networks to a wide range of images synthesized from segmentations during training, forcing them to generalize across contrasts. We show that networks trained within this framework generalize to a broad array of unseen MRI contrasts and surpass classical state-of-the-art brain registration accuracy by up to 12.4 Dice points for a variety of tested contrast combinations. Critically, training on arbitrary shapes synthesized from noise distributions results in competitive performance, removing the dependency on acquired data of any kind. Additionally, since anatomical label maps are often available for the anatomy of interest, we show that synthesizing images from these dramatically boosts performance, while still avoiding the need for real intensity images during training.

Keywords: Image registration, Brain, Magnetic resonance imaging (MRI)
Abstract: The application of deep learning approaches in medical image registration has decreased the registration time and increased registration accuracy. Most of the learning-based registration approaches considers this task as a one directional problem. As a result, only correspondence from the moving image to the target image is considered. However, in some medical procedures bidirectional registration is required. Here, we propose a registration framework with inverse consistency. The proposed method learns in an unsupervised manner a bidirectional transformation that approximates a diffeomorphism. We perform training and testing of the method on the publicly available LPBA40 MRI dataset and demonstrate its strong performance.

Keywords: Magnetic resonance imaging (MRI), Image registration, Brain
Abstract: Accurate registration of medical images is vital for doctor's diagnosis and quantitative analysis. In this paper, we propose a new unsupervised learning network SA-VoxelMorph for 3D deformable medical image registration. In this paper, we design a novel network architecture SAU-Net by introducing a new Binary Spatial Attention Module (BSAM) into skip connection of 3D U-Net, which can make full use of the spatial information extracted from the encoding and corresponding decoding stage. Moreover, from variational method, control function can better control the generation of registration field pmbphi. Therefore, we also propose a new registration loss function with novel smoothing term based on optimal control method to generate better pmbphi. We verify our method on two datasets including ADNI and PPMI, and obtain excellent results on magnetic resonance image (MRI) registration with higher average Dice scores and better diffeomorphic registration fields compared with other state-of-the-art methods. The experimental results show the method can achieve better performance in brain MRI registration.

Keywords: Image registration, Magnetic resonance imaging (MRI)
Abstract: Deformable image registration is widely utilized in medical image analysis, but most proposed methods fail in the situation of complex deformations. In this paper, we present a cascaded feature warping network to perform the coarse-to-fine registration. To achieve this, a shared-weights encoder network is adopted to generate the feature pyramids for the unaligned images. The feature warping registration module is then used to estimate the deformation field at each level. The coarse-to-fine manner is implemented by cascading the module from the bottom level to the top level. Furthermore, the multi-scale loss is also introduced to boost the registration performance. We employ two public benchmark datasets and conduct various experiments to evaluate our method. The results show that our method outperforms the state-of-the-art methods, which also demonstrates that the cascaded feature warping network can perform the coarse-to-fine registration effectively and efficiently.

Keywords: Image registration, Machine learning, Heart
Abstract: Registration between phases in 4D cardiac MRI is essential for reconstructing high-quality anatomy and appreciating the dynamics. Complex sequential compartmental motion and heterogeneous image quality make it challenging to design regularization functionals in classic optimization settings. In this study, we propose to introduce a novel motion representation model (MRM) into an image registration network to impose spatially variant prior for cardiac motion. A set of highly representative deformation vector fields (DVFs) were generated from high-contrast CTA images. In the form of a convolutional auto-encoder, the MRM was trained to capture the spatial variant pattern of the DVF Jacobian. The CT-derived MRM was then incorporated into an unsupervised network to facilitate 4D MRI registration. Our method was evaluated on ten 4D MRI scans with multi-compartment manual segmentations and achieved 2.25~mm target registration errors (TRE) on left ventricle. Compared to networks without MRM, introduction of the MRM reduced TREs on two ventricles and pulmonary artery with statistical significance. Compared to the tuned SimpleElastix, our method achieved comparable results on all compartments without statistical significance, but with a much shorter registration time of 0.02 s.

Keywords: Computer-aided detection and diagnosis (CAD), Whole-body
Abstract: Universal lesion detection in computed tomography (CT) images is an important yet challenging task due to the large variations in lesion type, size, shape, and appearance. Considering that data in clinical routine (such as the DeepLesion dataset) are usually annotated with a long and a short diameter according to the standard of Response Evaluation Criteria in Solid Tumors (RECIST) diameters, we propose RECIST-Net, a new approach to lesion detection in which the four extreme points and center point of the RECIST diameters are detected. By detecting a lesion as keypoints, we provide a more conceptually straightforward formulation for detection, and overcome several drawbacks (e.g., requiring extensive effort in designing data-appropriate anchors and losing shape information) of existing bounding-box-based methods while exploring a single-task, one-stage approach compared to other RECIST-based approaches. Experiments show that RECIST-Net achieves a sensitivity of 92.49% at four false positives per image, outperforming other recent methods including those using multi-task learning.

Keywords: Image segmentation, Magnetic resonance imaging (MRI)
Abstract: The lack of annotated data is a common problem in medical image segmentation tasks. In this paper, we present a novel multi-task semi-supervised segmentation algorithm with a curriculum-style learning strategy. The proposed method includes a segmentation task and an auxiliary regression task. Concretely, the auxiliary regression task aims to learn image-level properties such as the size and centroid position of target region to regularize the segmentation network, enforcing the pixel-level segmentation result match the distributions of these regressions. In addition, these regressions are treated as pseudo labels for the learning of unlabeled data. For the purpose of decreasing noise from the deviation of inferred labels, we adopt the inequality constraint for the learning of unlabeled data, which would generate a tolerance interval where the prediction within it would not be published to reduce the impact of prediction deviation of regression network. Experimental results on both 2017 ACDC dataset and PROMISE12 dataset demonstrate the effectiveness of our method.

Keywords: Image segmentation, Optimization method, Heart
Abstract: In developmental biology, quantification of cellular morphological features is a critical step to get insight into tissue morphogenesis. The efficacy of the quantification tools rely heavily on robust segmentation techniques which can delineate individual cells/nuclei from cluttered environment. Application of popular neural network methods is restrained by the availability of ground truth necessary for 3D nuclei segmentation. Consequently, we propose a convolutional neural net- work method, combined with graph theoretic approach for 3D nuclei segmentation of mouse embryo cardiomyocytes, imaged by light-sheet microscopy. The designed neural network architecture encapsulates both membrane and nuclei cues for 2D detection. A global association of the 2D nuclei detection is performed by solving a linear optimization with second order constraint to obtain 3D nuclei reconstruction.

Keywords: Image segmentation, Nerves, Microscopy - Light, Confocal, Fluorescence
Abstract: One of the critical steps in improving accurate single neuron reconstruction from three-dimensional (3D) optical microscope images is the neuronal structure segmentation. However, they are always hard to segment due to the lack in quality. Despite a series of attempts to apply convolutional neural networks (CNNs) on this task, noise and disconnected gaps are still challenging to alleviate with the neglect of the non-local features of graph-like tubular neural structures. Hence, we present an end-to-end segmentation network by jointly considering the local appearance and the global geometry traits through graph reasoning and a skeleton-based auxiliary loss. The evaluation results on the Janelia dataset from the BigNeuron project demonstrate that our proposed method exceeds the state-of-the-art algorithms in performance.

Keywords: Ultrasound, Tracking (time series analysis), Image segmentation
Abstract: We present a novel algorithm for needle tracking in ultrasound-guided needle insertion. Most previous research assumes that in ultrasound images the needle is a straight and bright line, but needles can bend due to the interaction with heterogeneous tissue. We utilize a novel weighted RANSAC curve fitting method combined with probabilistic Hough transform to track the curved needle robustly, and the algorithm can additionally utilize external tracking information, such as robotic kinematics, to further improve the tracking accuracy. We compared against classical tracking algorithms and a U-Net model, testing over different needle curvature and tissues. Our proposed algorithm achieves higher accuracy in tip location, shaft fitting, and tip angle. In-vivo porcine experiments with naturally bending short needles also show our method better tracked the tip location.

Keywords: Image segmentation, Microscopy - Light, Confocal, Fluorescence
Abstract: Morphological reconstruction of dendritic spines from fluorescent microscopy is a critical open problem in neuro-image analysis. Existing segmentation tools are ill-equipped to handle thin spines with long, poorly illuminated neck membranes. We address this issue, and introduce an unsupervised path prediction technique based on a stochastic framework which seeks the optimal solution from a path-space of possible spine neck reconstructions. Our method is specifically designed to reduce bias due to outliers, and is adept at reconstructing challenging shapes from images plagued by noise and poor contrast. Experimental analyses on two photon microscopy data demonstrate the efficacy of our method, where an improvement of 12.5% is observed over the state-of-the-art in terms of mean absolute reconstruction error.

Keywords: Vessels, Image segmentation, Machine learning
Abstract: The segmentation of the bronchus tree and pulmonary arter- ies and veins in CT scans plays an important role in patient care for both diagnosis and treatment phases. The extraction of theses tubular structures using either manual or interactive segmentation tools is time-consuming and prone to error due to the complexity of aerial and vascular trees. In this work, we propose a fully automatic method, relying on cascaded convolutional neural networks, to tackle lungs, bronchus and pulmonary arteries and veins segmentation. The first component based on a 2D U-Net architecture is dedicated to right and left lung segmentation. The second component relies on a three-paths 2.5D fully convolutional networks along axial, coronal and sagittal slices and focuses on tubular structures. Performance is assessed on a database of 193 chest CT scans.

Keywords: Histopathology imaging (e.g. whole slide imaging), Machine learning
Abstract: The global cancer burden is on the rise, and Artificial Intelligence (AI) has become increasingly crucial to achieve more objective and efficient diagnosis in digital pathology. Current AI-assisted histopathology analysis methods need to address the following two issues. First, the color variations due to use of different stains need to be tackled such as with stain style transfer technique. Second, in parallel with heterogeneity, datasets from individual clinical institutions are characterized by privacy regulations, and thus need to be addressed such as with robust data-private collaborative training. In this paper, to address the color heterogeneity problem, we propose a novel generative adversarial network with one orchestrating generator and multiple distributed discriminators for stain style transfer. We also incorporate Federated Learning (FL) to further preserve data privacy and security from multiple data centers. We use a large cohort of histopathology datasets as a case study.

Keywords: Ultrasound, Image enhancement/restoration(noise and artifact reduction), Machine learning
Abstract: Harmonic imaging’s popularity arises from its ability to produce high contrast resolution images. However, its need for at least two successive firings remains a hindering factor for a faster imaging process. In this work, a novel approach for ultrasound tissue harmonic imaging using a single firing is introduced utilizing deep learning concepts. This is achieved by implementing a network to predict the linear signal component output from a received nonlinear echo signal as input. Two different architectures were implemented: Convolutional AutoEncoder (CAE) and U-Net –like architecture. The dataset consists of 6k 3D focused K-wave simulations of multi scatterers varying in position, radius and speed of sound in a tissue-like medium with speckle noise. Each simulation is performed twice with the same tissue properties in a linear and a nonlinear environment. For each transmission, a transmission frequency of 7.5MHz was used and the acquired raw RF signals were sampled. The networks achieved a Mean Squared Error (MSE) value of 9.1x10-06 on the validation set between the linear ground truth signals and the predicted output. Moreover, the Total Harmonic Distortion (THD) value in the model’s predicted results is 1.615% compared to 31.75% in the nonlinear environment demonstrating an enhancement in harmonics suppression by 91.3%. Furthermore, the proposed technique is exploited in harmonic imaging by subtracting the predicted linear component from the received nonlinear echo to suppress the fundamental frequency. This harmonic imaging approach achieved an average THD of 119.5%, while the conventional Pulse Amplitude Modulation (PAM) method achieved 71.22% allowing a better harmonic to fundamental ratio. These results open the door for the implementation of harmonic imaging with a comparable quality to the conventional PAM technique, yet with an increased frame-rate and reduced motion artifacts.

Keywords: Image synthesis
Abstract: Automated segmentation of cell nuclei is used to analyze individual cells to determine the number of nuclei in a 3D volume. Deep learning approaches that segment nuclei require large amounts of annotated (ground truth) microscopy volumes for training. In many cases acquiring large amounts of annotated volumes may not be possible and data augmentation methods must be used. One approach has been the use of synthetic volumes for training. Alternate methods employ spherical and ellipsoidal nuclear models for synthetic ground truth generation, resulting in segmentation that does not accurately match nuclei morphology. In this paper, we present a technique to generate synthetic non-ellipsoidal nuclei microscopy images using Bezier Curves. We test our approach by training a modified 3D U-Net. Our results indicates that our synthetic non-ellipsoidal nuclei approach achieves improved segmentation on volumes with irregularly shaped nuclei.

Keywords: Classification, Machine learning, Pattern recognition and classification
Abstract: Hereditary hemolytic anemias are genetic disorders that affect the shape and density of red blood cells. Genetic tests currently used to diagnose such anemias are expensive and unavailable in the majority of clinical labs. Here, we propose a method for identifying hereditary hemolytic anemias based on a standard biochemistry method, called Percoll gradient, obtained by centrifuging a patient's blood. Our hybrid approach consists on using spatial data-driven features, extracted with a convolutional neural network and spectral handcrafted features obtained from fast Fourier transform. We compare late and early feature fusion with AlexNet and VGG16 architectures. AlexNet with late fusion of spectral features performs better compared to other approaches. We achieved an average F1-score of 88% on different classes suggesting the possibility of diagnosing of hereditary hemolytic anemias from Percoll gradients. Finally, we utilize Grad-CAM to explore the spatial features used for classification.

Keywords: fMRI analysis, Brain, Functional imaging (e.g. fMRI)
Abstract: It is difficult to distinguish schizophrenia (SZ) and bipolar disorder with psychosis (BPP) due to their overlapping symptoms. Indeed, there has been evidence supporting different subtypes within them. Data-driven clustering approaches are commonly used to explore biologically meaningful biotypes using neuroimaging features. However, previous studies typically consider pair-wise subject relationships. Here, we propose a hypergraph clustering method to explore biotypes. Our method extracts high-order features via hyperedges sampling, measures similarity and then regroups subjects using community detection. We applied it to identify biotypes of 100 BPP and 100 SZ patients using brain functional connectivity estimated from resting-state fMRI data, and compared with solutions from K-means and normalized cut (Ncut). Two reliable biotypes were identified and had greater differences in functional connectivity than groups determined by clinical diagnosis. Our method also outperformed K-means and Ncut for the clustering ability and computation efficiency. In summary, the proposed method is promising for developing biotypes, targeting accurate clinical diagnosis for psychosis.

Keywords: Image reconstruction - analytical & iterative methods, Machine learning, Computed tomography (CT)
Abstract: Current mainstream of CT reconstruction methods based on deep learning usually needs to fix the scanning geometry and dose level, which will significantly aggravate the training cost and need more training data for clinical application. In this paper, we propose a parameter-dependent framework (PDF) which trains data with multiple scanning geometries and dose levels simultaneously. In the proposed PDF, the geometry and dose level are parameterized and fed into two multi-layer perceptrons (MLPs). The MLPs are leveraged to modulate the feature maps of CT reconstruction network, which condition the network outputs on different scanning geometries and dose levels. The experiments show that our proposed method can obtain competing performance similar to the original network trained with specific geometry and dose level, which can efficiently save the extra training cost for multiple scanning geometries and dose levels.

Keywords: Breast, Machine learning, Classification
Abstract: Breast conserving surgery is a standard cancer treatment to resect breast tumors while preserving healthy tissue. The reoperation rate can be as high as 35% due to the difficulties associated with detection of remaining cancer in surgical margins. REIMS is a mass spectrometry method that can address this challenge through real-time measurement of molecular signature of tissue. However, the collection of breast spectra to train a cancer detection model is time consuming and large samples sizes are not practical. We propose an application of self-supervised learning to improve the performance of cancer detection at surgical margins using a limited number of labelled data samples. A deep model is trained for the intermediate task of capturing latent features of REIMS data without the use of cancer labels. The model compensates for the small data size by dividing the spectra into smaller patches and shuffling their order, generating new instances. By interrogating the shuffled data and learning the order of its patches, the model captures the characteristics of the data. The learnt weights from the model are then transferred to a subsequent network and fine-tuned for cancer detection. The proposed method achieved the accuracy, sensitivity and specificity to 97%, 91% and 100%, respectively, in data from 144 cancer and normal REIMS samples.

Keywords: Image segmentation, Magnetic resonance imaging (MRI), Image registration
Abstract: Landmark localisation in medical imaging has achieved great success using deep encoder-decoder style networks to regress heatmap images centered around the target landmarks. However, these networks are large and computationally expensive. Moreover, their clinical use often requires human interaction, opening the door for manual correction of low confidence predictions. We propose PHD-Net: a lightweight, multi-task, patch-based network combining heatmap and displacement regression. We design a simple candidate smoothing strategy to fuse its two-task outputs, generating the final prediction with quantified confidence. We evaluate PHD-Net on hundreds of Short Axis and Four Chamber cardiac MRIs, showing promising results.

Keywords: Ultrasound, Lung, Machine learning
Abstract: Lung ultrasound (LUS) imaging is used to assess lung abnormalities, including the presence of B-line artefacts due to fluid leakage into the lungs caused by a variety of diseases. However, manual detection of these artefacts is challenging. In this paper, we propose a novel methodology to automatically detect and localize B-lines in LUS videos using deep neural networks trained with weak labels. To this end, we combine a convolutional neural network (CNN) with a long short-term memory (LSTM) network and a temporal attention mechanism. Four different models are compared using data from 60 patients. Results show that our best model can determine whether one-second clips contain B-lines or not with an F1 score of 82.1%, and extracts a representative frame with B-lines with an accuracy of 87.5%.

Keywords: Multi-modality fusion, Brain, Machine learning
Abstract: Approaches involving the use of learning architectures on multimodal neuroimaging data tend to assume uniformity in the way information is stored in various sub-domains of the brain, thus not catering to the differences across functional and structural sub-domains. We introduce a learning framework to effectively incorporate multimodal features using structural and functional MRI data from a dataset of schizophrenia patients and controls, accounting for and exploiting the heterogeneity in the sub-domains of the brain. We analyze these sub-domains in terms of their functional interactions (i.e. within and between network connectivity) and structural properties (gray matter volume). By using Bayesian optimization on a search space of flexible multimodal architectures with multiple branches, we demonstrate that the discriminatory information from structural and functional sub-domains can be better recovered if the complexity of subspace structure in the model can be tuned to reflect the extent of non-linearity with which each sub-domain encodes the information. Our repeated cross-validated results from a schizophrenia classification problem show that for better classification and interpretation, sub-domains known for their role or disruption in Schizophrenia require more sophisticated subspace structure in the model compared to others. Our work emphasizes on the requirement to create multimodal frameworks that can adapt based on differences in the way various sub-domains of the brain encode discriminatory information. This is important to not only have better-performing prediction models but also to reveal sub-domains associated with the outcome at hand.

Keywords: Image segmentation, Magnetic resonance imaging (MRI), Bone
Abstract: Automatic segmentation of the musculoskeletal system in pediatric magnetic resonance (MR) images is a challenging but crucial task for morphological evaluation in clinical practice. We propose a deep learning-based regularized segmentation method for multi-structure bone delineation in MR images, designed to overcome the inherent scarcity and heterogeneity of pediatric data. Based on a newly devised shape code discriminator, our adversarial regularization scheme enforces the deep network to follow a learnt shape representation of the anatomy. The novel shape priors based adversarial regularization (SPAR) exploits latent shape codes arising from ground truth and predicted masks to guide the segmentation network towards more consistent and plausible predictions. Our contribution is compared to state-of-the-art regularization methods on two pediatric musculoskeletal imaging datasets from ankle and shoulder joints.

Keywords: Diffusion weighted imaging, Classification, Image segmentation
Abstract: In this paper, we provide a framework to evaluate new scalar quantities for higher order tensors (HOT) appearing in high angular resolution diffusion imaging (HARDI). These can potentially serve as biomarkers. It involves flattening of HOTs and extraction of the diagonal D-components. Experiments performed in the 4th order case reveal that D-components encode the geometric information unlike the isometric 6D 2nd order Voigt form. The existing invariants obtained from the Voigt form are considered for comparison. We also notice that D-components can be useful in segmentation of white matter structures in crossing regions and classification. Results on phantom and the synthetic dataset support the conclusions.

Keywords: Ultrasound, Heart, Image segmentation
Abstract: Bayesian Optimization (BO) is a well-studied hyperparameter tuning technique that is more efficient than grid search for high-cost, high-parameter machine learning problems. Echocardiography is a ubiquitous modality for evaluating heart structure and function in cardiology. In this work, we use BO to optimize the architectural and training-related hyperparameters of a previously published deep fully convolutional neural network model for multi-structure segmentation in echocardiography. In a fair comparison, the resulting model outperforms this recent state of the art on the annotated CAMUS dataset in both apical two- and four-chamber echo views. We report mean Dice overlaps of 0.95, 0.96, and 0.93 on left ventricular endocardium, epicardium, and left atrium respectively. We also observe significant improvement in derived clinical indices, including smaller median absolute errors for left ventricular end-diastolic volume (4.9ml vs. 6.7), end-systolic volume (3.1ml vs. 5.2), and ejection fraction (2.6% vs. 3.7); and much tighter limits of agreement, which were already within inter-rater variability for non-contrast echo. While these results demonstrate the benefits of BO for echocardiography segmentation over even a recent state-of-the-art framework, they must still be validated against large-scale independent clinical data.

Keywords: Image segmentation, Pattern recognition and classification, Machine learning
Abstract: We present a recurrent network for 3D reconstruction of neurons that sequentially generates binary masks for every object in an image with spatio-temporal consistency. Our network models consistency in two parts: (i) local, which allows exploring non-occluding and temporally-adjacent object relationships with bi-directional recurrence. (ii) non-local, which allows exploring long-range object relationships in the temporal domain with skip connections. Our proposed network is end-to-end trainable from an input image to a sequence of object masks, and, compared to methods relying on object boundaries, its output does not require post-processing. We evaluate our method on three benchmarks for neuron segmentation and achieved state-of-the-art performance on the SNEMI3D challenge.

Keywords: Image synthesis, Computed tomography (CT), Classification
Abstract: Detector-based spectral computed tomography is a recent dual-energy CT (DECT) technology that offers the possibility of obtaining spectral information. From this spectral data, different types of images can be derived, amongst others virtual monoenergetic (monoE) images. MonoE images potentially exhibit decreased artifacts, improve contrast, and overall contain lower noise values, making them ideal candidates for better delineation and thus improved diagnostic accuracy of vascular abnormalities.

In this paper, we are training convolutional neural networks~(CNNs) that can emulate the generation of monoE images from conventional single energy CT acquisitions. For this task, we investigate several commonly used image-translation methods. We demonstrate that these methods while creating visually similar outputs, lead to a poorer performance when used for automatic classification of pulmonary embolism (PE). We expand on these methods through the use of a multi-task optimization approach, under which the networks achieve improved classification as well as generation results, as reflected by PSNR and SSIM scores. Further, evaluating our proposed framework on a subset of the RSNA-PE challenge data set shows that we are able to improve the Area under the Receiver Operating Characteristic curve (AuROC) in comparison to a naïve classification approach from 0.8142 to 0.8420.

Keywords: Ultrasound, Optimization method, Quantification and estimation
Abstract: Quantitative ultrasound (QUS) aims at recovering quantitative properties of tissue microstructure by investigating the power spectra of the radio frequency data or statistics of the envelope of the backscattered signal. The accuracy and precision of microstructural properties are necessary for a correct description of tissue microstructure. In this paper, we propose a novel technique to analytically estimate the effective scatterer diameter and the acoustic concentration using simultaneous attenuation compensation. We show that our proposed technique substantially outperforms a recently-proposed, dynamic-programming based method, and is 100 times faster.

Keywords: Retinal imaging, Eye, Machine learning
Abstract: Retinopathy of prematurity (ROP) is one of the commonest causes of acquired blindness in children. The stage of ROP is an important step to evaluate the ROP severity for disease control and management. However, there are still various challenges for ROP stage since the pattern of ROP is relatively obscure compared to the entire fundus image. Also, the dataset is small and the image quality is quite poor. To address these issues, we develop a multi-instance learning (MIL) network, which can extract the features of the images and these features can be enhanced by a fully convolutional network (FCN). The spatial score map (SSM) produced by the FCN is cropped into small patches and fed into the proposed MIL for further feature learning. An attention mechanism is leveraged to guide the MIL pooling, which can focus on the ROP features of different stages and improve the staging results. The proposed network is evaluated on an in-house ROP dataset and experimental results demonstrate that our proposed method is promising for the stage of ROP.

Keywords: Microscopy - Light, Confocal, Fluorescence, Tracking (time series analysis), In-vivo cellular and molecular imaging
Abstract: Cell tracking in time-lapse microscopy images is important to study biological processes. We propose a new multi-object tracking method based on a dynamic memory network and template matching. Cells are detected by a fully convolutional neural network, and multiple dynamic memory units are used to track cells in successive frames. The template is dynamically updated using a long-short term memory with attention to cope with changing cell appearance. Our method includes a motion constraint based on cell motion statistics to improve the robustness. To handle cell mitosis events, a deep mitosis detector is integrated in our tracking method. We evaluated the proposed method on time-lapse microscopy datasets including data from the Cell Tracking Challenge. Experimental results demonstrate that our method yields state-of-the-art results or better results than baseline methods.

Keywords: Multi-modality fusion, Machine learning, Whole-body
Abstract: Images in the medical domain are fundamentally different from the general domain images. Consequently, it is infeasible to directly employ general domain Visual Question Answering (VQA) models for the medical domain. Additionally, medical image annotation is a costly and time-consuming process. To overcome these limitations, we propose a solution inspired by self-supervised pretraining of Transformer-style architectures for NLP, Vision, and Language tasks. Our method involves learning richer medical image and text semantic representations using Masked Vision-Language Modeling as the pretext task on a large medical image+caption dataset. The proposed solution achieves new state-of-the-art performance on two VQA datasets for radiology images – VQA-Med 2019 and VQA-RAD, outperforming even the ensemble models of previous best solutions. Moreover, our solution provides attention maps which help in model interpretability.

Keywords: Histopathology imaging (e.g. whole slide imaging), Classification, Tissue
Abstract: Cervical cancer is the most common gynecological disease, which seriously endangers women’s health. Histopathological diagnosis is the most reliable determinant for cervical cancer prognosis and treatment. However, it is a time- consuming and error-prone task for pathologists to manually analyze the whole slide image(WSI). Automated detection of cancer tissue and histopathological diagnosis are of great signification to reduce the workload for pathologists. So we proposed a method combining deep learning(DL) and traditional machine learning(ML) to realize the rapid analysis of the cervical cancer WSI. Our method first applied a patch- level network to detect the pathological patches and generate a heatmap that can highlight the tumor area. Considering the difficulty of obtaining pixel-level annotations, we designed a weak annotation strategy for training the network and verified the feasibility of this strategy on a public dataset. Besides, we used overlapping heatmap generation strategy(Overlapping- HMGS) and features sharing strategy of fully convolutional neural networks(CNN) to balance the accuracy and speed of tumor region detection. Then features extracted from the heatmaps are fed into a traditional machine learning model for the WSI-level prediction. Our approach not only achieved excellent performance with 97% accuracy on our cervical cancer WSI dataset, but it is also efficient and interpretable.

Keywords: Image segmentation, Skin
Abstract: We propose a new residual block for convolutional neural networks and demonstrate its state-of-the-art performance in medical image segmentation. We combine attention mechanisms with group convolutions to create our group attention mechanism, which forms the fundamental building block of our network, FocusNet++. We adapt a hybrid loss based on balanced cross entropy, Tversky loss and the adaptive logarithmic loss to create an overall loss function for fast convergence to a high-performance solution. Our results show that FocusNet++ achieves state-of-the-art results across various benchmark metrics for the ISIC 2018 melanoma segmentation and the cell nuclei segmentation datasets with fewer parameters and FLOPs.

Keywords: Machine learning, Image segmentation, Brain
Abstract: Typical machine learning frameworks heavily rely on an underlying assumption that training and test data follow the same distribution. In medical imaging which increasingly begun acquiring datasets from multiple sites or scanners, this identical distribution assumption often fails to hold due to systematic variability induced by site or scanner dependent factors. Therefore, we cannot simply expect a model trained on a given dataset to consistently work well, or generalize, on a dataset from another distribution. In this work, we address this problem, investigating the application of machine learning models to unseen medical imaging data. Specifically, we consider the challenging case of Domain Generalization (DG) where we train a model without any knowledge about the testing distribution. That is, we train on samples from a set of distributions (sources) and test on samples from a new, unseen distribution (target). We focus on the task of white matter hyperintensity (WMH) prediction using the multi-site WMH Segmentation Challenge dataset and our local in-house dataset. We identify how two mechanically distinct DG approaches, namely domain adversarial learning and mix-up, have theoretical synergy. Then, we show drastic improvements of WMH prediction on an unseen target domain.

Keywords: Image segmentation, Machine learning, Brain
Abstract: Neural networks for automated image segmentation are typically trained to achieve maximum accuracy, while less attention has been given to the calibration of their confidence scores. However, well-calibrated confidence scores provide valuable information towards the user. We investigate several post hoc calibration methods that are straightforward to implement, some of which are novel. They are compared to Monte Carlo (MC) dropout and are applied to neural networks trained with cross-entropy (CE) and soft Dice (SD) losses on BraTS 2018 and ISLES 2018. Surprisingly, models trained on SD loss are not necessarily less calibrated than those trained on CE loss. In all cases, at least one post hoc method improves the calibration. There is limited consistency across the results, so we can't conclude on one method being superior. In all cases, post hoc calibration is competitive with MC dropout. Although average calibration improves compared to the base model, subject-level variance of the calibration remains similar.

Keywords: X-ray imaging, Computed tomography (CT), Image reconstruction - analytical & iterative methods
Abstract: This paper introduces a single-scan dual-energy coded aperture computed tomography system that enables material characterization at a reduced exposure level. Rapid kVp switching with a single-static block/unblock coded aperture relies on coded illumination with a plurality of X-ray spectra created by the kVp switching. Based on the tensor representation of the projection data, an algorithm to estimate the missing measurements in the tensor is proposed. This results in a full set of synthesized measurements that can be used with filtered back-projection or iterative reconstruction algorithms to accurately reconstruct the object in each energy channel. Simulation results validate the effectiveness of the proposed cost-effective solution to attain material characterization in low-dose dual-energy CT.

Keywords: Machine learning, Brain, Magnetic resonance imaging (MRI)
Abstract: Over the past years, deep learning proved its effectiveness in medical imaging for diagnosis or segmentation. Nevertheless, to be fully integrated in clinics, these methods must both reach good performances and convince area practitioners about their interpretability. Thus, an interpretable model should make its decision on clinical relevant information as a domain expert would. With this purpose, we propose a more interpretable classifier focusing on the most widespread autoimmune neuroinflammatory disease: multiple sclerosis. This disease is characterized by brain lesions visible on MRI (Magnetic Resonance Images) on which diagnosis is based. Using Integrated Gradients attributions, we show that the utilization of brain tissue probability maps instead of raw MR images as deep network input reaches a more accurate and interpretable classifier with decision highly based on lesions.

Keywords: Computer-aided detection and diagnosis (CAD), Machine learning, Pattern recognition and classification
Abstract: Radiomics has shown a remarkable ability for different types of cancers such as glioma to predict the clinical outcome. It can have a non-invasive means of evaluating the immunotherapy response prior to treatment. However, the use of deep convolutional neural networks (CNNs)-based radiomics requires large training sets. To resolve this problem, we propose a new type of radiomic features that model distribution with a Gaussian mixture model (GMM) of learned 3D CNN features. Using these deep radiomic features (DRFs), we aim to predict the immune marker status (low versus high) and overall survival for glioma patients. We extract the DRFs by aggregating the activation maps of a pre-trained 3D-CNN within labeled tumor regions of MRI scans that corresponded immune markers of 151 patients. Our experiments are performed to assess the relationship between the proposed DRFs, three immune cell markers (Macrophage M1, Neutrophils and T Cells Follicular Helper), and measure their association with overall survival. Using the random forest (RF) model, DRFs was able to predict the immune marker status with area under the ROC curve (AUC) of 78.67, 83.93 and 75.67% for Macrophage M1, Neutrophils and T Cells Follicular Helper, respectively. Combined the immune markers with DRFs and clinical variables, Kaplan-Meier estimator and Log-rank test achieved the most significant difference between predicted groups of patients (short-term versus long-term survival) with p = 4.31 10-7 compared to p = 0.03 for Immune cell markers, p = 0.07 for clinical variables , and p = 1.45 10-5 for DRFs. Our results suggest that the proposed DRFs used with a RF classifier can significantly consider prognosticating patients with brain tumour prior to surgery via routinely-acquired imaging data.

Keywords: Machine learning, Pattern recognition and classification, Image registration
Abstract: Organ morphology is a key indicator for prostate disease diagnosis and prognosis. For instance, In longitudinal study of prostate cancer patients under active surveillance, the volume, boundary smoothness and their changes are closely monitored on time-series MR image data. In this paper, we describe a new framework for forecasting prostate morphological changes, as the ability to detect such changes earlier than what is currently possible may enable timely treatment or avoiding unnecessary confirmatory biopsies. In this work, an efficient feature-based MR image registration is first developed to align delineated prostate gland capsules to quantify the morphological changes using the inferred dense displacement fields (DDFs). We then propose to use kernel density estimation (KDE) of the probability density of the DDF-represented textit{future morphology changes}, between current and future time points, before the future data become available. The KDE utilises a novel distance function that takes into account morphology, stage-of-progression and duration-of-change, which are considered factors in such subject-specific forecasting. We validate the proposed approach on image masks unseen to registration network training, without using any data acquired at the future target time points. The experiment results are presented on a longitudinal data set with 331 images from 73 patients, yielding an average Dice score of 0.865 on a holdout set, between the ground-truth and the image masks warped by the KDE-predicted-DDFs.

Keywords: Machine learning, X-ray imaging, Lung
Abstract: The shortage of Radiologists is inspiring the development of Deep Learning (DL) based solutions for detecting cardio, thoracic and pulmonary pathologies in Chest radiographs through multi-institutional collaborations. However, sharing the training data across multiple sites is often impossible due to privacy, ownership and technical challenges. Although Federated Learning (FL) has emerged as a solution to this, the large variations in disease prevalence and co-morbidity distributions across the sites may hinder proper training. We propose a DL architecture with a Convolutional Neural Network (CNN) followed by a Graph Neural Network (GNN) to address this issue. The CNN-GNN model is trained by modifying the Federated Averaging algorithm. The CNN weights are shared across all sites to extract robust features while separate GNN models are trained at each site to leverage the local co-morbidity dependencies for multi-label disease classification. The Chexpert dataset is partitioned across five sites to simulate the FL set up. Federated training did not show any significant drop in performance over centralized training. The site specific GNN models also demonstrated their efficacy in modelling local disease co-occurrence statistics leading to an average area under the ROC curve of 0.79 with a 1.74% improvement.

Keywords: Image segmentation, Machine learning, Microscopy - Light, Confocal, Fluorescence
Abstract: We present SplineDist, an instance segmentation convolutional neural network for bioimages extending the popular StarDist method. While StarDist describes objects as star-convex polygons, SplineDist uses a more flexible and general representation by modelling objects as planar parametric spline curves. Based on a new loss formulation that exploits the properties of spline constructions, we can incorporate our new object model in StarDist's architecture with minimal changes. We demonstrate in synthetic and real images that SplineDist produces segmentation outlines of equal quality than StarDist with smaller network size and accurately captures non-star-convex objects that cannot be segmented with StarDist.

Keywords: X-ray imaging, Bone, Image enhancement/restoration(noise and artifact reduction)
Abstract: Quality of cone-beam computed tomography (CBCT) images are marred by artifacts in the presence of metallic implants. Metal artifact correction is a challenging problem in CBCT scanning especially for large metallic objects. The appearance of artifacts also change greatly with the body part being scanned. Metal artifacts are more pronounced in orthopedic imaging, when metals are in close proximity of other high density materials, such as bones. Recently introduced mask incorporating deep learning networks for metal inpainting showed improvements over classical methods in CBCT image quality. However, generalization of results for more than one body part is still not investigated. We investigate, the use of gated convolutions for mask guidance inpainting to improve the filling of the corrupt metal area in projection domain. The neural network was trained with eight clinical metal affected datasets by incorporating data augmentation techniques. In the end, we validate our method on six clinical datasets. Our method shows promising results both in projections and reconstructed images.

Keywords: Inverse methods, Machine learning, Other-modality
Abstract: Light Field Microscopy (LFM) is an imaging technique that captures 3D spatial information in a single 2D image. LFM is attractive because of its relatively simple implementation and fast acquisition rate. However, classic 3D reconstruction typically suffers from high computational cost, low lateral resolution, and reconstruction artifacts. In this work, we propose a new physics-based learning approach to improve the performance of the reconstruction under realistic conditions, these being lack of training data, background noise, and high data dimensionality. First, we propose a novel description of the system using a linear convolutional neural network. This description is complemented by a method that compacts the number of views of the acquired light field. Then, this model is used to solve the inverse problem under two scenarios. If labelled data is available, we train an end-to-end network that uses the Learned Iterative Shrinkage and Thresholding Algorithm (LISTA). If no labelled data is available, we propose an unsupervised technique that uses only unlabelled data to train LISTA by making use of Wasserstein Generative Adversarial Networks (WGANs). We experimentally show that our approach performs better than classic strategies in terms of artifact reduction and image quality.

Keywords: Magnetic resonance imaging (MRI), Machine learning, Image reconstruction - analytical & iterative methods
Abstract: The main focus of this work is a novel framework for the joint reconstruction and segmentation of parallel MRI (PMRI) brain data. We introduce an image domain deep network for calibrationless recovery of undersampled PMRI data. The proposed approach is the deep-learning (DL) based generalization of local low-rank based approaches for uncalibrated PMRI recovery including CLEAR [1]. Since the image domain approach exploits additional annihilation relations compared to k-space based approaches, we expect it to offer improved performance. To minimize segmentation errors resulting from undersampling artifacts, we combined the proposed scheme with a segmentation network and trained it in an end-to-end fashion. In addition to reducing segmentation errors, this approach also offers improved reconstruction performance by reducing overfitting; the reconstructed images exhibit reduced blurring and sharper edges than independently trained reconstruction network.

Keywords: Machine learning, Image reconstruction - analytical & iterative methods, Magnetic resonance imaging (MRI)
Abstract: Bilinear models such as low-rank and compressed sensing, which decompose the dynamic data to spatial and temporal factors, are powerful and memory efficient tools for the recovery of dynamic MRI data. These methods rely on sparsity and energy compaction priors on the factors to regularize the recovery. Motivated by deep image prior, we introduce a novel bilinear model, whose factors are regularized using convolutional neural networks. To reduce the run time, we initialize the CNN parameters by with pre-training them on pre-acquired data with longer acquisition time. Since fully sampled data is not available, the pretraining was performed on undersampled data in an unsupervised fashion. We use sparsity regularization of the network parameters to minimize the overfitting of the network to measurement noise. Our experiments on free-breathing and ungated cardiac CINE data acquired using a navigated golden-angle gradient-echo radial sequence show the ability of our method to provide reduced spatial blurring as compared to low-rank and SToRM reconstructions.

Keywords: Diffusion weighted imaging, Machine learning, Magnetic resonance imaging (MRI)
Abstract: In addition to the more traditional diffusion tensor imaging (DTI), over time, reconstruction techniques like HARDI have been proposed, which have a comparatively higher scanning time due to increased measurements, but are significantly better in the estimation of fiber structures. In order to make HARDI-based analysis faster, we propose an approach to reconstruct more HARDI volumes in q-space. The proposed GAN-based architecture leverages several modules, including a multi-context module, feature inter-dependencies module along-with numerous losses such as L1, adversarial, and total variation loss, to learn the transformation. The method is backed by some encouraging quantitative and visual results.

Keywords: Image segmentation, Vessels, Computed tomography (CT)
Abstract: Accurate three-dimensional (3D) segmentation of the coronary artery lumen is an essential step in the quantitative analysis of coronary artery stenosis. However, due to the small size and complex structure of the coronary artery tree, it is difficult and laborious to perform voxel-by-voxel labeling of the lumen on 3D cardiac computed tomography angiography (CCTA) images. Since radiologists tend to focus only on the regions of interest, the annotations collected are often imperfect and contain some false-negative targets. To address the problem of partial annotations, a 3D convolutional neural network (CNN)-based approach for coronary artery lumen segmentation is proposed. Our CNN model adopts an U-Net like backbone and has multiple auxiliary branches on the expanding path. In the inference stage, an uncertain map of the network prediction is generated from the multiple abstract feature maps, which is used to refine the segmentation results. Experimental results on the MICCAI 2020 Automated Segmentation of Coronary Arteries (ASOCA) challenge dataset show that our method achieves even better segmentation accuracy with partial annotations than the backbone model trained with full annotations.

Keywords: Machine learning, Cells & molecules, Computational Imaging
Abstract: Mitosis identification and localization gives us information on cell status and behaviour, and is foundational to many biological research and medical applications. In this paper, we propose a novel two-stage computer vision framework for the automation of mitosis event detection in large-scale time-lapse phase contrast microscopy image sequences. First stage is a two dimensional image segmentation method for proposing probable spatial locations of mitotic events. The second stage is spatio-temporal mitotic localization refinement driven image sequence classification. Second stage analyzes the temporal dynamics augmented with visual contextual learning and is specifically trained for classifying true mitotic events from hard negative cases wherein the cells appear very similar to mitotic cells. To the best of our knowledge, the proposed method outperforms all previous approaches of mitosis key-point detection. Our state of the art approach achieved top performance in the CVPR 2019 contest on mitosis detection in phase contrast microscopy image sequences with an average F1 score of 0.8804 on the test data of C2C12-16 dataset.

Keywords: Breast, Computer-aided detection and diagnosis (CAD), Machine learning
Abstract: Breast cancer screening with mammography is the most efficient way to reduce breast cancer mortality. However, the large population and the use of double reading creates a high workload that heavily burdens the efficiency of the radiologists. Hence, Computer-Aided Detection (CAD) systems are being developed to assist the radiologists. In this study, an algorithm for the automatic detection and classification of breast masses, based on temporal subtraction of sequential mammograms, image registration and machine learning, is presented. While, some previous studies in the literature proposed temporal analysis by creating a new feature vector, temporal subtraction takes into consideration the entire prior image. A new dataset, consisting of 40 cases (two time points and two views of each breast per patient, a total of 160 images), with precisely annotated mass locations was created. The accuracy of the classification of masses as benign or suspicious increased from 85.7% (using temporal analysis) to 92.9% (using temporal subtraction). The improvement was statistically significant with p < 0.05. These results demonstrate the effectiveness of temporal subtraction for the diagnosis of breast masses.

Keywords: Image segmentation, Connectivity analysis, Microscopy - Light, Confocal, Fluorescence
Abstract: Recent advances in imaging technologies for generating large quantities of high-resolution 3D images, especially multispectral labeling technology such as Brainbow, permits unambiguous differentiation of neighboring neurons in a densely labeled brain. This enables, for the first time, the possibility of studying the connectivity between many neurons from a light microscopy image. The lack of reliable automated neuron morphology reconstruction, however, makes data analysis the bottleneck of extracting rich informatics in neuroscience. Supervoxel-based neuron segmentation methods have been proposed to solve this problem, however, previous approaches have been impeded by the large numbers of errors which arise in the final segmentation. In this paper, we present a novel unsupervised approach to trace neurons from multispectral Brainbow images, which prevents segmentation errors and tracing continuity errors using two innovations: First, we formulate a Gaussian mixture model-based clustering strategy to improve the separation of segmented color channels that provides accurate skeletons for the next steps. Then, a skeleton graph approach is proposed to allow the identification and correction of discontinuities in the neuron tree topology. We find that these innovations allow better performance over current state-of-the-art approaches, which results in more accurate neuron tracing results close to human expert annotation.

Keywords: Machine learning, Pattern recognition and classification, Magnetic resonance imaging (MRI)
Abstract: Medical segmentation is performed to determine the bounds of regions of interest (ROI) prior to surgery. By allowing the study of growth, structure, and behaviour of the ROI in the planning phase, critical information can be obtained, increasing the likelihood of a successful operation. Usually, segmentations are performed manually or via machine learning methods trained on manual annotations. In contrast, this paper proposes a fully automatic, unsupervised inpainting-based brain tumour segmentation system for T1-weighted MRI. First, a deep convolutional neural network (DCNN) is trained to reconstruct missing healthy brain regions. Then, upon application, anomalous regions are determined by identifying areas of highest reconstruction loss. Finally, superpixel segmentation is performed to segment those regions. We show the proposed system is able to segment various sized and abstract tumours and achieves a mean and standard deviation Dice score of 0.771 and 0.176, respectively.

Keywords: Computer-aided detection and diagnosis (CAD), Machine learning, Classification
Abstract: The placement of a central venous catheter (CVC) for venous access is a common clinical routine. Nonetheless, various clinical studies report that CVC insertions are unsuccessful in up to 20% of all cases. Among other, typical complications include the incidence of a pneumothorax, hemothorax, arterial puncture, venous air embolism, arrhythmias or catheter knotting. In order to detect the CVC tip in chest X-ray (CXR) images, and to evaluate the catheter placement, we propose a HRNet-based key point detection approach in combination with a probabilistic constellation model. In a cross-validation study, we show that our approach not only enables the exact localization of the CVC tip, but also of relevant anatomical landmarks. Moreover, the probabilistic model provides a likelihood score for tip position which allows us to identify malpositioned CVCs.

Keywords: Image acquisition, Optical coherence tomography, Surgical guidance/navigation
Abstract: Optical coherence tomography (OCT) is a near-infrared light based imaging modality that enables depth scans with a high spatial resolution. By scanning along the lateral dimensions, high-resolution volumes can be acquired. This allows to characterize tissue and precisely detect abnormal structures in medical scenarios. However, its small field of view (FOV) limits the applicability of OCT for medical examinations. We therefore present an automated setup to move an OCT scan head over arbitrary surfaces. By mounting the scan head to a highly accurate robot arm, we obtain precise information about the position of the acquired volumes. We implement a geometric approach to stitch the volumes and generate the surface scans. Our results show that a precise stitching of the volumes is achieved with mean absolute errors of 0.078mm and 0.098mm in the lateral directions and 0.037mm in the axial direction. We can show that our setup provides automated surface scanning with OCT of samples and phantoms larger than the usual FOV.

Keywords: Image segmentation, Retinal imaging, Eye
Abstract: Diabetic retinopathy (DR) is one of the most common microvascular complications of diabetes. Early and accurate screening of DR from fundus images is crucial for the ophthalmologist to make treatment plans. In recent years, many deep learning-based methods have been proposed for medical image segmentation. However, the DR lesions segmentation still meets great challenges. In this work, we propose a novel cycle adaptive multi-target weighting network (CAMWNet) that mimics the biological vision system of the human brain. The network consists of two major parts: a novel adaptive multi-target weighting network (AMWNet) for DR lesions segmentation and a reverse data recovery network (RRN) to simulate the cycle perception in visual hierarchy. In addition, a novel joint loss function is designed to optimize the CAMWNet. Comprehensive experiments on Indian Diabetic Retinopathy Image Dataset (IDRiD) show that, CAMWNet achieves better performance than other state-of-the-art methods with accuracy, Dice similarity coefficients, sensitivity and specificity of 98.33%, 53.84%, 48.54% and 99.88%, respectively.

Keywords: Machine learning, Brain, Magnetic resonance imaging (MRI)
Abstract: Magnetic resonance imaging (MRI) has a potential for early diagnosis of individuals at risk for developing Alzheimer’s disease (AD). Cognitive performance in healthy elderly people and in those with mild cognitive impairment (MCI) has been associated with measures of cortical gyrification [1] and thickness (CT) [2], yet the extent to which sulcal measures can help to predict AD conversion above and beyond CT measures is not known. Here, we analyzed 721 participants with MCI from phases 1 and 2 of the Alzheimer’s Disease Neuroimaging Initiative, applying a two-state Markov model to study the conversion from MCI to AD condition. Our preliminary results suggest that MRI-based cortical features, including sulcal morphometry, may help to predict conversion from MCI to AD.

Keywords: Image segmentation, Microscopy - Light, Confocal, Fluorescence, Cells & molecules
Abstract: We present a novel method for model-based segmentation of extended, blob-like objects on curved surfaces. Our method addresses several challenges arising when imaging curved biological membrane, such as out-of-membrane signal and geometry-induced background variations. We use a particle-based reconstruction of the membrane geometry, moment-conserving intensity interpolation from pixels to surface particles, and model-based in-surface segmentation. Our method denoises and deconvolves images, corrects for background variations, and quantifies the number, size, and intensity of segmented objects. We benchmark the accuracy of the method and present two applications to (1) neuroepithelial focal adhesion sites during optic cup morphogenesis in zebrafish and (2) reconstituted membrane domains bearing the small GTPase Rab5 on spherical beads.

Keywords: Machine learning, Image reconstruction - analytical & iterative methods, Bone
Abstract: 3D image reconstruction is a common basis of medical image analysis, which requires a sequence of 2D slices/tomograms obtained from the relative motion to provide enough 3D information. When considering only the task to localize exception objects, a pair of two-view perspective 2D images may also be able to provide enough 3D information, which, however, has not been well studied. In this paper, we proposed the concept of Ultimate Reconstruction (UR) that reconstructs a 3D image from only a pair of two-view perspective 2D images. We resort techniques of generative adversarial network (GAN) to deal with this task, where we propose the Sense-consistency GAN (SGAN) with the sense-consistency constraint to learning the potential coarse-to-fine sense information during training the generative model. Experiments on the KiTS19 dataset with 300 subjects demonstrate that our SGAN achieves MAE / SSIM / PSNR values of 11.16% / 66.50% / 23.82 when using only two 2D perspective images. It supports the possibility of UR and indicates that SGAN is promising to deal with UR.

Keywords: Image synthesis, Magnetic resonance imaging (MRI), Brain
Abstract: Contrast-enhanced medical images offer vital insights for the accurate diagnosis, characterization and treatment of tumors, and are routinely used worldwide. Acquiring such images requires to inject the patient intravenously with a gadolinium-based contrast agent (GBCA). Although GBCAs are considered safe, recent concerns about their accumulation in the body tilted the medical consensus towards a more parsimonious usage. Focusing on the case of brain magnetic resonance imaging, this paper proposes a deep learning method that synthesizes virtual contrast-enhanced T1 images as if they had been acquired after the injection of a standard 0.100 mmol/kg dose of GBCA, taking as inputs complementary imaging modalities obtained either after a reduced injection at 0.025 mmol/kg or without any GBCA involved. The method achieves a competitive structural similarity index of 94.2%. Its asymptotic performance is estimated, and the most important input modalities are identified.

Keywords: Image reconstruction - analytical & iterative methods, Brain, Magnetic resonance imaging (MRI)
Abstract: Whole-brain surface extraction is an essential topic in medical imaging systems as it provides neurosurgeons with a broader view of surgical planning and abnormality detection. To solve the problem confronted in current deep learning skull stripping methods lacking prior shape information, we propose a new network architecture that incorporates knowledge of signed distance fields and introduce an additional Laplacian loss to ensure that the prediction results retain shape information. We validated our newly proposed method by conducting experiments on our brain magnetic resonance imaging dataset (111 patients). The evaluation results demonstrate that our approach achieves comparable dice scores and also reduces the Hausdorff distance and average symmetric surface distance, thus producing more stable and smooth brain isosurfaces.

Keywords: Multi-modality fusion, Lung, Computed tomography (CT)
Abstract: Fusion of multimodal data is important for disease understanding. In this paper, we propose a new method of fusion exploiting the uncertainty in prediction produced by the individual modality learners. Specifically, we extend the joint label fusion method by taking model uncertainty into account when estimating correlations among predictions produced by different modalities. Through experimental study in survival prediction for non-small cell lung cancer patients who received surgical resection, we demonstrated promising performance produced by the proposed method.

Keywords: Heart, Nuclear imaging (e.g. PET, SPECT), Image reconstruction - analytical & iterative methods
Abstract: Dose reduction in cardiac SPECT perfusion imaging is of great clinical importance owing to its potential radiation risks. In this study, we investigate the benefit of using full-dose data processed by deep learning (DL) as a learning target for improving the accuracy of reduced-dose studies. We demonstrated this approach in the experiments with a set of 895 clinical cases, in which we employed a pre-trained DL model for obtaining the full-dose target used for training a denoising network on reduced-dose images. The quantitative results show that the proposed approach could further improve the detection accuracy of perfusion defects (using the non-prewhitening matched filter as a numerical observer) on 50% dose studies over that of training directly with full-dose reconstruction. In addition, there was no loss observed in the spatial resolution of the reconstructed left ventricular wall as measured by its full-width at half-maximum.

Keywords: Ultrasound, Image acquisition, Machine learning
Abstract: In this paper, a single-probe ultrasonic imaging system that captures multi-variable quantitative profiles is presented. As pathological changes cause biomechanical property variation, quantitative imaging has great potential for lesion characterization. The proposed system simultaneously extracts four clinically informative quantitative biomarkers, such as the speed of sound, attenuation, effective scatter density, and effective scatter radius, in real-time using a single scalable neural network. The performance of the proposed system was evaluated through numerical simulations, and phantom and ex vivo measurements. The simulation results demonstrated that the proposed SQI-Net reconstructs four quantitative images with PSNR and SSIM of 19.52 dB and 0.8251, respectively, while achieving a parameter reduction of 75% compared to the design of four parallel networks, each of which was dedicated to a single parameter. In the phantom and ex vivo experiments, the SQI-Net demonstrated the classification of cyst, and benign- and malignant-like inclusions through a comprehensive analysis of four reconstructed images.

Keywords: Image segmentation, Machine learning, Magnetic resonance imaging (MRI)
Abstract: Generative adversarial networks using a cycle-consistency loss facilitate unpaired training of image-translation models and thereby exhibit a very high potential in manifold medical applications. However, the fact that images in one domain potentially map to more than one image in another domain (e.g. in case of pathological changes) exhibits a major challenge for training the networks. In this work, we offer a solution to improve the training process in case of many-to-one mappings by modifying the cycle-consistency loss. We show formally and empirically that the proposed method improves the performance significantly without radically changing the architecture and without increasing the overall complexity. We evaluate our method on thigh MRI scans with the final goal of segmenting the muscle in fat-infiltrated patients' data.

Keywords: Histopathology imaging (e.g. whole slide imaging), Breast, Computer-aided detection and diagnosis (CAD)
Abstract: The advent of digital pathology presents opportunities for computer vision for fast, accurate, and objective solutions for histopathological images and aid in knowledge discovery. This work uses deep learning to predict genomic biomarkers - TP53 mutation, PIK3CA mutation, ER status, PR status, HER2 status, and intrinsic subtypes, from breast cancer histopathology images. Furthermore, we attempt to understand the underlying morphology as to how these genomic biomarkers manifest in images. Since gene sequencing is expensive, not always available, or even feasible, predicting these biomarkers from images would help in diagnosis, prognosis, and effective treatment planning. We outperform the existing works with a minimum improvement of 0.02 and a maximum of 0.13 AUROC scores across all tasks. We also gain insights that can serve as hypotheses for further experimentations, including the presence of lymphocytes and karyorrhexis. Moreover, our fully automated workflow can be extended to other tasks across other cancer subtypes.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Brain
Abstract: The amount of biomedical data continues to grow rapidly. However, the ability to analyze these data is limited due to privacy and regulatory concerns. Machine learning approaches that require data to be copied to a single location are hampered by the challenges of data sharing. Federated Learning is a promising approach to learn a joint model over data silos. This architecture does not share any subject data across sites, only aggregated parameters, often in encrypted environments, thus satisfying privacy and regulatory requirements. Here, we describe our Federated Learning architecture and training policies. We demonstrate our approach on a brain age prediction model on structural MRI scans distributed across multiple sites with diverse amounts of data and subject (age) distributions. In these heterogeneous environments, our Semi-Synchronous protocol provides faster convergence.

Keywords: Magnetic resonance imaging (MRI), Brain, Image segmentation
Abstract: Most publicly available brain MRI datasets are very homogeneous in terms of scanner and protocols, and it is difficult for models that learn from such data to generalize to multi-center and multi-scanner data. We propose a novel data augmentation approach with the aim of approximating the variability in terms of intensities and contrasts present in real world clinical data. We use a Gaussian Mixture Model based approach to change tissue intensities individually, producing new contrasts while preserving anatomical information. We train a deep learning model on a single scanner dataset and evaluate it on a multi-center and multi-scanner dataset. The proposed approach improves the generalization capability of the model to other scanners not present in the training data.

Keywords: Magnetic resonance imaging (MRI), Prostate, Image quality assessment
Abstract: No-reference image quality assessment in magnetic resonance (MR) imaging is a challenging task due to the variable nature of these images and lack of standard quantification methods, which makes the interpretation to be almost always subjective. In this study, we propose an architecture where we: (i) extended the no-reference image quality assessment problem of MRI into a full-reference image quality assessment using unpaired generative adversarial network (GAN) and (ii) employed a weakly-supervised trained deep classifier to determine the quality of MR images by comparing each image with its synthetic higher quality reference image. Using this approach, we achieved 11.28% improvement in the accuracy of our MR image quality assessment algorithm on an independent data test with FPR in detecting low quality images, reduced from 13% to 9.6%.

Keywords: Magnetic resonance imaging (MRI), Brain, Other-method
Abstract: Brain morphometry performed with magnetic resonance (MR) imaging is affected by partial volume (PV) effects when single voxels contain the signal from two different tissues. This paper proposes a generalization of the MP2RAGE sequence PV estimation model which accounts for transmitted magnetic field (B1+) inhomogeneities at 7T. Our simulation experiments demonstrated that the PV estimation error of the proposed model is significantly lower than the error obtained with the same model neglecting B1+ inhomogeneities (p<0.0001). The accuracy and precision of the B1+ model (acc=92.0%, prec=89.6%) was significantly increased compared to the non B1+ model (acc=69.8%, prec=65.4%). This highlights the importance of accounting for B1+ inhomogeneities when computing PV on MP2RAGE data, which would otherwise limit the accuracy of brain morphometry at 7T.

Keywords: Magnetic resonance imaging (MRI), Image enhancement/restoration(noise and artifact reduction), Machine learning
Abstract: Accelerated MRI acquisitions by taking fewer samples in the k-space involve a trade-off between the image-quality degradation due to artifacts and acquisition time. The purpose of our study was to reduce artifacts in reconstructing MR images from under-sampled k-space data in fast-acquisition MRI. In this study, we proposed a novel massive-training artificial neural network (MTANN) scheme coupled with a k-space-sampling-pattern-specific kernel to improve the image quality by reducing artifacts while preserving the anatomic structures. We conducted experiments to evaluate the performance of our scheme with under-sampled MR images of 20 patients with 795 slices. The results showed that our proposed MTANN scheme reduced artifacts in reconstructed MR images from under-sampled k-space data substantially, while the image quality was well-maintained.

Keywords: Magnetic resonance imaging (MRI), Image reconstruction - analytical & iterative methods, Computational Imaging
Abstract: Due to the capability of fast multi-parametric quantitative imaging, magnetic resonance fingerprinting has become a promising quantitative magnetic resonance imaging (QMRI) approach. However, the highly undersampled and noise-contaminated k-space data will cause critical spatial artifacts, which subsequently lead to inaccurate estimation of the quantitative parameters. In this paper, we introduce a novel structured low-rank matrix recovery and subspace modeling framework (SLRSM) to recover temporal MRF data from its highly undersampled and noisy Fourier coefficients. Numerical experiments demonstrate that the proposed algorithm is capable of providing improved accuracy for parameter estimation over the state-of-the-art algorithms.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Prostate
Abstract: The utility of high-quality imaging of Prostate Cancer (PCa) using 3.0 Tesla MRI (versus 1.5 Tesla) is well established, yet a vast majority of MRI units across many counties are 1.5 Tesla. Recently, Deep Learning has been applied successfully to augment radiological interpretation of medical images. However, training such models requires very large amount of data, and often the models do not generalize well to data with different acquisition parameters. To address this, we introduce domain standardization, a novel method that enables image synthesis between domains by separating anatomy- and modality-related factors of images. Our results show an improved PCa classification with an AUC of 0.75 compared to traditional transfer learning methods. We envision domain standardization to be applied as a promising tool towards enhancing the interpretation of lower resolution MRI images, reducing the barriers of the potential uptake of deep models for jurisdictions with smaller populations.

Keywords: Image reconstruction - analytical & iterative methods, Inverse methods, Magnetic resonance imaging (MRI)
Abstract: Deep neural networks have been successfully applied to a wide variety of inverse problems arising in biomedical imaging. These networks are often trained using a forward model, which is not only used to generate the training data but is often incorporated directly into the network itself. However, these approaches lack robustness to misspecification of the forward model: if at test time the forward model varies (even slightly) from the one the network was trained on, the network performance can degrade substantially. Given a network trained to solve an initial image reconstruction problem with a known forward model, we propose novel retraining procedures that adapts the network to reconstruct measurements from a perturbed forward model. Our procedures do not require access to more labeled data (i.e., ground truth images), but only a relatively small collection of measurements under the perturbed forward model. We demonstrate this simple retraining procedure empirically achieves robustness to changes in the forward model for image reconstruction in magnetic resonance imaging from undersampled k-space measurements.

Keywords: Image segmentation, Vessels, X-ray imaging
Abstract: The segmentation of coronary arteries by convolutional neural network is promising yet requires a large amount of labor intensive manual annotations. Transferring knowledge from retinal vessels in widely-available public labeled fundus images (FIs) has a potential to reduce the annotation requirement for coronary artery segmentation in X-ray angiograms (XAs) due to their common tubular structures. However, it is challenged by the cross-anatomy domain shift due to the intrinsically different vesselness characteristics in different anatomical regions under even different imaging protocols. To solve this problem, we propose a semi-supervised cross-anatomy domain adaptation (SS-CADA) which requires only limited annotations for coronary arteries in XAs. With the supervision from a small number of labeled XAs and publicly available labeled FIs, we propose a vesselness-specific batch normalization (VSBN) to individually normalize feature maps for them considering their different cross-anatomic vesselness characteristics. In addition, to further facilitate the annotation efficiency, we employ a self-ensembling mean-teacher (SE-MT) to exploit abundant unlabeled XAs by imposing a prediction consistency constraint. Extensive experiments show that our SS-CADA is able to solve the challenging cross anatomy domain shift, achieving accurate segmentation for coronary arteries given only a small number of labeled XAs.

Keywords: Machine learning, Pattern recognition and classification, X-ray imaging
Abstract: Recent advances in deep learning have led to an accurate diagnosis of pneumonia from chest X-ray images. However, these models usually require large labeled training datasets, not always available in practice. Furthermore, combining images from different medical centers does not preserve the accuracy of the results mainly because of differences in image acquisition settings. In this work, we propose an approach aiming to overcome this challenge, consisting of a subspace-based domain adaptation technique to increase pneumonia detection accuracy using a small training dataset. This dataset is augmented with automatically selected images from a large dataset acquired in a different medical center. This is performed by computing a subspace basis of the target domain dataset on which is projected the source dataset to find the most representative images. Augmenting the training set using the proposed method allows achieving an improvement from 90.03% to 96.18% in overall accuracy using the Xception neural network.

Keywords: X-ray imaging, Machine learning
Abstract: In this paper we address the task of unsupervised domain adaptation for multi-label classification problems with convolutional neural networks. We particularly consider the domain shift in between X-ray data sets. Domain adaptation between different X-ray data sets is especially of practical and clinical importance to guarantee applicability across hospitals and clinics, which may use different machines for image captioning. In contrast to the usual multi-class setting, in multi-label classification tasks multiple labels can be assigned to an input instance instead of just one label. While most related work focus on domain adaptation for multi-class tasks, we consider the more general case of multi-label classification across domains. We propose an adversarial domain adaptation approach, in which the discriminator is equipped with additional conditional information regarding the current classification output. Our experiments show promising and competitive results on publicly available data sets, compared to state of the art approaches.

Keywords: Integration of multiscale information, Atlases, Whole-body
Abstract: Recently, we presented the registration of a whole-body cellular expression atlas to a high-resolution electron microscopy (EM) dataset, automatic segmentation of all cell somata and nuclei, and clustering of the cells according to gene expression or morphological parameters [1]. This dataset is a valuable resource containing rich biological information. However, the size and complexity of this holistic resource (currently 231 image sources adding up to 2.02 TB lossless compressed image data, including EM, light microscopy, segmentation images and segment feature tables) poses a challenge to its effective interrogation for scientific discovery. To integrate this dataset we developed MoBIE: a free and open-source platform for multi-modal big image data exploration and sharing. MoBIE (https://github.com/mobie/mobie) consists of an object store backend for cloud based image data sharing, GitHub based storage of tabular data, and an easy to install Fiji [3] plugin for integrated browsing of the whole dataset (Fig. 1). Thanks to lazy-loading even TB-sized datasets can be smoothly explored on a standard computer. It soon became apparent that, in addition to [1], also other datasets including various modalities such as EM tomography [5] can be efficiently shared using this platform. In fact, within few months MoBIE transformed the way we integrate and share multi-modal big image data at our institute. We are therefore excited to present this platform as we hope that it will facilitate holistic image data exploration and sharing also at other institutions.

[1] Vergara et al., “Whole-body integration of gene expression and single-cell morphology”, bioRxiv, https://doi.org/10.1101/2020.02.26.961037, 2020 [2] Schindelin et al., “Fiji: an open-source platform for biological-image analysis”, 2012, Nat. Methods 9, 676–682. [3] Pietzsch et al., “BigDataViewer: visualization and processing for large image data sets”, 2015, Nat. Methods 12, 481–483. [4] Schmid et al., "A high-level 3D visualization API for Java and ImageJ", 2010, BMC Bioinformatics, 11(1): 1 [5] Cortese et al., “Integrative Imaging Reveals SARS-CoV-2-Induced Reshaping of Subcellular Morphologies”, Cell Host & Microbe, 2020, 28, 853-866.e5

Keywords: Microscopy - Intravital, Microscopy - Electron, Animal models and imaging
Abstract: Brain metastases (BM) are a devastating and frequent consequence of cancer, including malignant melanoma, triple negative and HER2-positive breast and lung cancer. Although current therapies prolong survival, treatment options are currently limited. Our work focusses on understanding the underlying mechanisms of BM formation and unravelling key interactions between the cancer cells and the brain microenvironment. Hereto, we perform intravital microscopy (IVM), which uniquely allows us to study BM growth long-term with high temporal resolution. Moreover, correlating IVM to three-dimensional electron microscopy (3D EM) enables gaining unique insights into the critical events that result in brain colonisation.

Keywords: Integration of multiscale information, Microscopy - Super-resolution, X-ray imaging
Abstract: Imaging of biological matter across resolution scales entails the challenge of preserving the direct and unambiguous correlation of subject features from the macroscopic to the microscopic level. In this talk we present in detail the computer vision tasks part of a correlative imaging platform developed on a synchrotron (Beamline B24, Diamond Light Source, UK) specifically for imaging cells in 3D under cryogenic conditions by using X-rays and visible light. An integrated, user-friendly platform for 3D correlative imaging of cells in vitreous ice by using super-resolution structured illumination microscopy in conjunction with soft X-ray tomography was developed. The accurate matching of structures in both modalities required an analysis workflow with several intermediate images where error accumulation needs to be quantitatively estimated because of the important difference of scale and the unknown aspect of structure under study. This approach was demonstrated by studying the process of reovirus release from intracellular vesicles during the early stages of infection and identifying intracellular virus-induced structures. We present here the different steps of the workflow used, where the analysis was performed using ec-CLEM, and we detail the challenges encountered and the need for further developments.

Keywords: Microscopy - Electron, Microscopy - Light, Confocal, Fluorescence, Image registration
Abstract: AIM AND CHALLENGE: Rapidly changing features in any biological sample are challenging to efficiently trap and image by electron microscopy (EM). The free-living nematode C. elegans is widely used to study embryonic development, yet the fast kinetics of cell division hampers targeting specific developmental stages for ultrastructural study. C. elegans embryos are also encased in a chitinaceous shell, creating a diffusion barrier that precludes traditional aldehyde-based fixation. Therefore, one must trap and image these transient embryonic structures, then locate and subsequently capture the same volume by an appropriate EM technique, all while minimizing a variety of artifacts.

EXPERIMENTAL APPROACH: Here we performed high-pressure-freezing (HPF) of spatially constrained C. elegans worms in appropriate cryoprotectants and screened for embryonic cells by in situ cryo-fluorescence microscopy. Select worms were freeze substituted, resin embedded and further prepared to successfully locate and image the targeted cells by focused ion beam scanning electron microscopy (FIB-SEM).

METHODS AND RESULTS: Gravid C. elegans worms expressing chromatin (histone H2B-mCherry) and ER (signal peptide SP12-GFP) markers were collected in cellulose capillaries and placed in HPF planchettes. 15% dextran + 5% BSA (inside capillary) and perfluorodecalin filler (outside) reduced autofluorescence and provided cryoprotection. After HPF, frozen worms were imaged by cryo-fluorescence microscopy (Zeiss Examiner upright LSM 710 w/ Airyscan, Linkam cryostage with planchette holder). A bright, condensed H2B signal with low surrounding SP12 indicated embryos in metaphase; these samples were subjected to in situ quick freeze substitution and resin embedding. The cured resin was detached from the planchette and sectioned until the worm was revealed. Overlays of toluidine blue stained sections and cryo-fluorescence images dictated the region to be imaged by room temperature FIB-SEM. A 5 x 5 x 15 nm image stack was acquired, and the reconstructed 3-D image volume revealed the targeted embryo in metaphase.

By enabling cryo-fluorescence microscopy of thick samples, our workflow can be used to trap and image transient structures in model organisms in a near-native state, and then reconstruct their corresponding cellular architectures at high resolution and in 3-D by correlative volume EM.

Keywords: Machine learning, Image acquisition
Abstract: Multimodal imaging for histopathology has become a popular research field. Optical modalities such as fluorescence imaging and multiphoton imaging have been used in many histopathological studies. However, these studies are usually limited to small scales of samples due to the complexity of the imaging experiments, the extra-long time to extensively imaging the samples, and the intensive labor for annotating the samples. We present a smart multimodal histopathological slide scanning system powered by machine learning and the Python-based microscope control and imaging processing linking package Pycro-manager [1]. The system can automatically scan, analyze, and detect malignant regions on a slide and perform multiphoton imaging at different magnifications on the detected areas. The state-of-the-art weakly-supervised learning model used to detect the lesion regions can be trained with unannotated slides [2]. Our system can minimize the need for user intervention and increase the throughput for complex multimodal optical imaging experiments.

Keywords: Microscopy - Super-resolution
Abstract: The ability to image biological specimens through an optical system dates back to the middle ages and enabled important discoveries such as the first visualization of microbes or of chromosomes within cells. The last two decades has seen several revolutions in optical microscopy that are starting to reveal, for instance, how microbes within complex colonies can work together to predate on other microbes, or how chromosomes are organized within complex organisms. Particularly, these new microscopies are also allowing us to link structure with biological function at multiple scales. In my presentation, I will review these recent microscopy developments and demonstrate with practical examples how they are being used to change the way we understand biological

Keywords: Microscopy - Super-resolution, Probabilistic and statistical models & methods, Heart
Abstract: Single molecule localization microscopy has been a primary mode of resolving the clustering patterns of proteins, particularly membrane bound receptors and ion channels. A notable example is the clustering of ryanodine receptor (RyR) calcium (Ca2+) channels in cardiac muscle. With protocols such as DNA-PAINT, we can now localize and count individual channels within these clustered arrays at a spatial resolution ≤ 10 nm. However, with such data, the functional correlates of these nanoscale structures are often missing. We recently developed an experimental protocol to acquire and register correlative image data of RyR clustering morphologies and the fast Ca2+ signals (bursts of cytoplasmic Ca2+ called ‘Ca2+ sparks’) produced by the local ensembles of these clusters. The protocol included two-dimensional (2D) TIRF microscopy recordings of sub-plasmalemmal Ca2+ sparks of living cardiomyocytes isolated enzymatically from adult rats. Cells were then fixed and subjected DNA-PAINT of the ryanodine receptor within the same regions. The Ca2+ and the DNA-PAINT images were then registered using a semi-automated protocol described recently. The localized nature of Ca2+ sparks (widths ranging between 1-6 μm) and the punctate appearance of resolved RyR antibody tags lent themselves to the automated detection of their respective centroids using tailored 2D-Gaussian match filter detection protocols. This discretization allowed us to both visualize and perform spatial statistics of the number of RyRs detected underneath the footprint of every Ca2+ spark. It also allowed us to observe that the Ca2+ sparks were recorded in a spatially non-random pattern featuring ‘hot spots’ where Ca2+ sparks spontaneously recurred over time. Using a Voronoi tessellation analysis we mapped these hot spots and correlated them with the DNA-PAINT maps of RyR channels.Local examination of the RyR patterns underneath each Ca2+ spark revealed a steep correlation between the size of the Ca2+ sparks and the underlying channel ensembles which typically ranged from ~ 5 to 100 channels. A weaker correlation was observed in cardiomyocytes isolated from rats with right ventricular failure, reflecting dysfunction in the local regulation of the RyRs in such pathologies. Animal experiments were conducted according to UK Animals (Scientific Procedures) Act with UK Home Office approval.

Keywords: Microscopy - Super-resolution, Single cell & molecule detection
Abstract: How do proteins organize in cells to fulfill their designated function? Proteins have a limited operating range and, consequently, their activity occurs at the nanometer scale, which requires advanced microscopy techniques to study associated processes.

Single-molecule localization microscopy (SMLM) relies on the temporally separated emission of individual molecules. While the acquired image is still diffraction limited, due to their sparsity, each molecule can be localized at nanometer precision and their coordinates accumulated to form a super-resolved image. But the power of SMLM goes far beyond generating images. Computational analysis tools allow extracting the spatial organization, colocalization, clustering and dynamics of individual molecules, which has transformed the study of nanoscale biology over the past decade.

I will introduce SMLM while discussing practical considerations as well as advantages and disadvantages. Along with a number of biological studies from cell and infection biology, I will then highlight the power of SMLM to understand subcellular structures, their organization and dynamics.

Contact: nanoinfection.org

Keywords: Microscopy - Super-resolution, In-vivo cellular and molecular imaging, Tracking (time series analysis)
Abstract: In recent years and with the help of advanced fluorescence microscopy techniques, immense progress has been made in characterizing and quantifying the behavior of single cells on the basis of molecular interactions and assemblies in the complex environment of live cultures. Importantly, single-molecule imaging enables the in vivo determination of the stoichiometry and molecular architecture of subcellular structures, yielding detailed, quantitative, spatiotemporally resolved molecular maps and unraveling dynamic heterogeneities and subpopulations on the subcellular level. Behind todays attractive super-resolved images and analyses hides a rather high complexity of in large detail tailored experimental designs for specific organisms and environments. We cannot answer our research questions about the in-situ behavior of molecular processes at a single molecules’ spatiotemporal resolution without highly optimized and robust tools - which are still largely missing for many biological research fields. In this talk, I will introduce some of our recently developed experimental and analytical tools alongside with our specific biological questions. If you are interested in a beginners guide on how to vamp up your wide field fluorescence microscope and sample preparations to single-molecule sensitivity, including many tips and tricks from my groups work, this is the right talk for you. I will also explain some of our advanced tools such as quantitative dual-color PALM imaging using dual fluorescent protein labeling in living cells, more detailed sample preparations like easy but precise drift correction by red-shifted beads or tracking of dense, highly dynamic single-molecule data.

References [1] Turkowyd, B., Virant, D., & Endesfelder, U. (2016). From single molecules to life: microscopy at the nanoscale. Analytical and bioanalytical chemistry, 408(25), 6885-6911. [2] Vojnovic, I., Winkelmeier, J., & Endesfelder, U. (2019). Visualizing the inner life of microbes: practices of multi-color single-molecule localization microscopy in microbiology. Biochemical Society Transactions, 47(4), 1041-1065. [3] Endesfelder, U. (2019). From single bacterial cell imaging towards in vivo single-molecule biochemistry studies. Essays in biochemistry, 63(2), 187-196.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Genes
Abstract: Access to large cohorts is key to obtaining the statistical power to detect even subtle (disease) effects and to generate robust results. However, assembling a large cohort is time consuming and expensive, and a common strategy is to pool data across different research centers. There are two opposing concepts for combining data: meta-analysis and mega analysis.

The Enhancing NeuroImaging Genetics through Meta-Analysis (ENIGMA) consortium has embraced the meta-analysis framework to conduct imaging genetics research. Moreover, in recent years dozens of disease working groups have formed within ENIGMA aiming to consolidate and analyze imaging data for specific disorders.

In brief, meta-analysis is a very common framework in genetics research: each study center analyzes the locally collected raw data and shares only the results of their analysis. Results are combined to obtain the final study result. This approach ensures that subject-level data are not shared (avoids the completion of time-intensive data share agreements) and individual centers can conveniently contribute their data to the common cause (low entry burden). However, the analysis pipeline (steps and software versions; quality control) has to be agreed upfront and executed in the same way by all participating sites causing some organizational overhead.

In contrast, mega analyses pool the raw data and perform a single analysis on the entire dataset giving it greater flexibility in the analysis design, however, individual-level have to be shared and thus raising a centers entry burden.

No matter which overall study design is chosen, with imaging data there are further challenges being faced by collaborative consortia, the most prominent being the harmonization of imaging data across sites and more generally addressing site-specific biases. This is a very active area of research. The talk will illustrate the different study designs and challenges as experienced typically faced by ENIGMA disease working groups.

Keywords: Data Mining, Multi-modality fusion, Parallel computing
Abstract: Federated Learning (FL) is an attractive solution for secured analysis of distributed data in multi-centric studies. FL is based on a decentralized learning paradigm which avoids sharing raw data across centres: while models are trained locally on the available data, only model parameters are subsequently shared to define an aggregated global model. The application of FL to typical biomedical data analysis scenarios is currently limited by several factors. First, model aggregation is often not possible when datasets are heterogeneous, for example when classes and views are not uniformly represented across data centres (i.e. non-iid distributed). Second, although FL avoids data sharing across centers, sharing model parameters may still open up the possibility of information leakage and privacy breaking in presence of malicious clients. Finally, there is currently a limited availability of production ready FL schemes that can be readily used in real-life multi-centric data analysis applications.

The goal of the project Fed-BioMed consists in developing a comprehensive platform for FL in healthcare applications. Fed-BioMed is currently relying on the federated learning library PySyft for extending numerical operations and tensor manipulation to the FL paradigm. The project is based on state-of-the-art FL approaches, while adopting optimization schemes robust to data heterogeneity among different centers. In particular, we are currently developing probabilistic approaches to FL based on a generative model of data and parameters variability across centres. This allows us to quantify and account for the discrepancy of data and model’s distributions across participants. Moreover, this kind of approach naturally allows to account for missing features and modalities in each centre. Furthermore, Fed-BioMed is conceived to ensure robustness to the problem of malicious attacks in FL through the development of opportune defense and monitoring strategies (e.g. privacy breaking and adversarial attacks, free-riders).

Fed-BioMed will be deployed on a consortium of partners from the ENIGMA study, which will allow the joint modeling of multiple datasets worldwide. Fed-BioMed is an open-source project: code, documentation and tutorials are (and will be) accessible at the link https://fedbiomed.gitlabpages.inria.fr.

This work is compliant with the ISBI ethical requirements.

Keywords: Probabilistic and statistical models & methods, Machine learning, Magnetic resonance imaging (MRI)
Abstract: Neuroscience has now truly transitioned into the era of big data and is witnessing an explosion of the number and types of biological measures that can -and are- measured in clinical populations. This has given rise to the emerging field of imaging epidemiology, which holds enormous potential to improve prediction of disease states in many clinical conditions. This involves moving beyond group averages (e.g. using classifiers to discriminate patients with brain disorders from healthy controls). Instead, what is necessary to understand variation in brain-derived measures across the population and to understand how such variation relates to health and disease at the level of the individual. However, to achieve this objective it is necessary to solve a number of key technical challenges, including scaling computational methods to very large samples, whilst appropriately dealing with many sources of variation, for example related to scanner variation, nuisance variation and sampling bias. Here, I will first show that the prevalent methods employed in the field do not properly accommodate such variation. I will then present work from our group that combines advanced Bayesian hierarchical models with asymptotically exact sampling methods to properly deal with such variation. I will then show applications of these methods to neuroimaging data derived from large population-based cohorts in order to derive biomarkers that predict disease state and outcome in psychiatric disorders. While the approaches I will discuss are fully generic, I will focus on normative modelling or ‘brain growth charting’ methods that provide the ability to capture clinically relevant variation in the imaging data without needing to assume a priori structure, such as clusters in the data. These methods provide a major step that will move the field beyond simple case-control comparisons, toward a better understanding of brain disorders and will pave the way toward precision medicine in psychiatry.

Keywords: Classification, Heart, Machine learning
Abstract: Artificial intelligence and machine learning have gained a renewed popularity thanks the recent advances in information technology such as the Internet of Things. This powerful technology helps make better decisions and accurate predictions in many domains including healthcare. In particular, Neural Networks (NN) can support medical workers to analyse patients’ data and quickly diagnose a particular disease such as heart arrhythmia. Nowadays, heart arrhythmia can be detected at early stages with the help of smart wearable devices that can record electric heart activities using Electro-Cardiograms (ECG) data.

Nevertheless, ECG data is considered as very sensitive. Given the recent data breach scandals, stakeholders face increasing challenges with ensuring data privacy guarantees and compliance with the GDPR. Therefore, there is an urgent need for tools enabling the protection of such data while still being able to launch predictive analytics and hence improve individuals’ lives. Under the PAPAYA project (www.papaya-project.eu) we address privacy concerns when analytics are performed by untrusted third-party processors. Since these tasks may be performed obliviously on protected data (i.e. encrypted data), the PAPAYA project develops dedicated privacy preserving modules that enable stakeholders to extract valuable information from this protected data, while being accurate. Among the operations, we focus on NN classifications and aim at addressing privacy concerns raised by the analysis of ECG data for arrhythmia detection. Our goal is to enable service providers (data processors) perform classification without discovering the input (the ECG data). We propose to combine the use of NN with secure two-party computation(2PC). Since 2PC protocols cannot efficiently support all operations, we propose to revisit the underlying NN operations and design a new, customized NN model that can be executed to classify arrhythmia accurately, and this, without disclosing the input ECG data to the service provider. This solution is described in our publication in FPS2019. This research study was conducted retrospectively using human subject data made available in open access by (https://www.physionet.org/physiobank/database/mitdb). Ethical approval was not required as confirmed by the license attached with the open access data.

Keywords: Machine learning, Histopathology imaging (e.g. whole slide imaging), Pattern recognition and classification
Abstract: Digital pathology tasks have greatly benefited from modern deep learning algorithms. However, their need for large quantities of annotated data has been identified as a key challenge. This need for data can be countered by using unsupervised learning methods in situations where data are abundant but access to annotations is limited. Feature representations learned from unannotated data using contrastive predictive coding have been shown to enable classifiers to obtain state of the art performance from relatively small amounts of annotated computer vision data. We present a modification to the contrastive predictive coding framework for use with digital pathology patches. This is achieved by introducing an alternative mask for building the latent context and using a multi-directional PixelCNN autoregressor. To demonstrate our proposed method we learn feature representations from the Patch Camelyon histology dataset. We show that our proposed modification can yield improved deep classification of histology patches.

Keywords: Tracking (time series analysis), Integration of multiscale information, Ultrasound
Abstract: In ultrasound video, the tracking and segmentation of carotid plaques are prerequisite for the analysis of plaque properties. The problem at hand is quite challenging as some issues such as low ultrasound image resolution, large variation between frames and low tracking efficiency need to addressed. Our method, Siamese automatic tracking and segmentation network (SiamATSN), is an end-to-end deep learning method. Multiple dual attention region proposal network (DARPN) blocks are developed to integrate multi-level features for better target detection. The DARPN block is composed of spatial-wise attention module, channel-wise attention module, and Region Proposal Network. Moreover, a fusion module is further introduced to capture long-range contextual clues. At last, concat module which combines low and high resolution features is embedded in the decoder path to attain a more precise segmentation. Extensive experiments are conducted and the experimental results showed that our approach outperforms the state-of-the-art deep learning based methods.

Keywords: Ultrasound, Machine learning, Heart
Abstract: The left ventricular ejection fraction (LVEF) is one of the most commonly measured cardiac parameters in echocardiography (echo). While the clinical guidelines suggest that apical views are used for LVEF assessment, these views are difficult to obtain for less experienced operators. Specifically, in point-of-care imaging, parasternal views are commonly used for rapid assessment of LVEF, since those views can be easier to obtain. However, robust LVEF estimation is challenging due to high variability of echo quality and cardiovascular structures across different patients. In this paper, we formulate a Bayesian deep learning approach for fully automatic LVEF estimation based on segmentation of the left ventricle (LV) in parasternal short-axis papillary muscles (PSAX-PM) level. The proposed approach exploits the LV segmentation uncertainty to improve the robustness of the reported LVEF. The experiments using a dataset of 2,680 patients show that the proposed approach could increase the LVEF estimation's R2 score with a noticeable margin of 17.9%, while automatically detecting and discarding around 6% of cases with the highest predictive uncertainty.

Keywords: Machine learning, Image segmentation, Brain
Abstract: Over the last decade, deep learning methods have achieved state-of-the-art for medical image segmentation tasks. However, the difficulty of obtaining sufficient labeled data can be a bottleneck. To this end, we design a novel active learning framework specially adapted to the brain tumor segmentation. Our approach includes a novel labeling cost designed to capture radiologists' practical labeling costs. This is combined with two acquisition functions to incorporate uncertainty and representation information, ensuring that the active learning selects informative and diverse data. The resulting procedure is a constrained combinatorial optimization problem. We propose an efficient algorithm for this task and demonstrate the proposed method's advantages for segmenting brain MRI data.

Keywords: Magnetic resonance imaging (MRI), Computer-aided detection and diagnosis (CAD), Muscle
Abstract: Muscle texture analysis in Magnetic Resonance Imaging (MRI) has revealed a good correlation with typical histological changes resulting from neuromuscular disorders. In this research, we assess the effectiveness of several features in describing intramuscular texture alterations in cases of Collagen VI-related myopathy. A T1-weighted Turbo Spin-Echo MRI dataset was used (N_subj = 26), consisting of thigh scans from subjects diagnosed with Ullrich Congenital Muscular Dystrophy or Bethlem Myopathy, with different severity levels, as well as healthy subjects. A total of 355 texture features were studied, including attributes derived from the Gray-Level Co-occurrence Matrix, the Run-Length Matrix, Wavelet and Gabor filters. The extracted features were ranked using the Support Vector Machine Recursive Feature Elimination (SVM-RFE) algorithm with Correlation Bias Reduction, prior to cross-validated classification with a Gaussian kernel SVM.

Keywords: Endoscopy, Gastrointestinal tract, Machine learning
Abstract: Place recognition in colonoscopy is needed for various reasons. 1) If a certain region needs to be rechecked during an endoscopy, the endoscopist needs to re-localize the camera accurately to the region of interest. 2) Place recognition is needed for same-patient follow-up colonoscopy to localize the region where a polyp was cut off. 3) Recent development in colonoscopic 3D reconstruction needs place recognition to establish long-range correspondence, e.g., for loop closure. However, traditional image retrieval techniques do not generalize well in colonic images. Moreover, although place recognition or instance-level image retrieval is a widely researched topic in computer vision and several benchmarks have been published for it, there has been no specific research or benchmarks in endoscopic images, which are significantly different from common images used in traditional computer vision tasks. In this paper we present a testing dataset with manually labeled groundtruth which comprises 10126 images from 20 colonoscopic subsequences. We perform an extensive evaluation on different existing place recognition techniques using different metrics.

Keywords: Optical coherence tomography, Eye, Computer-aided detection and diagnosis (CAD)
Abstract: Primary angle-closure glaucoma (PACG) is a major sub-type of glaucoma that is responsible for half of the glaucoma-related blindness worldwide. The early detection of PACG is very important, so as to provide timely treatment and prevent potential irreversible vision loss. Clinically, the diagnosis of PACG is based on the evaluation of anterior chamber angle (ACA) with anterior segment optical coherence tomography (AS-OCT). To this end, the Angle closure Glaucoma Evaluation (AGE) challenge held on MICCAI 2019 aims to encourage researchers to develop automated systems for angle closure classification and scleral spur (SS) localization. We participated in the competition and won the championship on both tasks. In this paper, we share some ideas adopted in our entry of the competition, which significantly improve the accuracy of scleral spur localization. There are extensive literatures on keypoint detection for the tasks such as human body keypoint and facial landmark detection. However, they are proven to fail on dealing with scleral spur localization in the experiments, due to the gap between natural and medical images. In this regard, we propose a set of constraints to encourage a two-stage keypoint detection framework to spontaneously exploit diverse information, including the image-level knowledge and contextual information around SS, from the AS-OCT for the accurate SS localization. Extensive experiments are conducted to demonstrate the effectiveness of the proposed constraints.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Image quality assessment
Abstract: Quality control (QC) is a vital step for all scientific data analyses and is critically important in the biomedical sciences. Image segmentation is a common task in medical image analysis, and automated tools to segment many regions from human brain MRIs are now well established. However, these methods do not always give anatomically correct labels. Traditional methods for QC tend to reject statistical outliers, which may not necessarily be inaccurate. Here, we make use of a large database of over 12,000 brain images that contain 68 parcellations of the human cortex, each of which was assessed for anatomical accuracy by a human rater. We trained three machine learning models to determine if a region was anatomically accurate (as ‘pass’, or ‘fail’) and tested the performance on an independent dataset. We found excellent performance for the majority of labeled regions. This work will facilitate more anatomically accurate large- scale multi-site research.

Keywords: Image enhancement/restoration(noise and artifact reduction), Computed tomography (CT), Lung
Abstract: While quantitative image features (radiomic) can be employed as informative indicators of disease progression, they are sensitive to variations in acquisition and reconstruction. Prior studies have demonstrated the ability to normalize heterogeneous scans using per-pixel metrics (e.g., mean squared error) and qualitative reader studies. However, the generalizability of these techniques and the impact of normalization on downstream tasks (e.g., classification) have been understudied. We present a multi-pronged evaluation by assessing image normalization techniques using 1) per-pixel image quality and perceptual metrics, 2) variability in radiomic features, and 3) task performance differences using a machine learning (ML) model. We evaluated a previously reported 3D generative adversarial network-based (GAN) approach, investigating its performance on low-dose computed tomography (CT) scans acquired at a different institution with varying dose levels and reconstruction kernels. While the 3D GAN achieved superior metric results, its impact on quantitative image features and downstream task performance did not result in universal improvement. These results suggest a more complicated relationship between CT acquisition and reconstruction parameters and their effect on radiomic features and ML model performance, which are not fully captured using per-pixel metrics alone. Our approach provides a more comprehensive picture of the effect of normalization.

Keywords: Image segmentation, Microscopy - Electron, Connectivity analysis
Abstract: We propose a new topology-preserving method for 3D image segmentation. We treat the image as a stack of 2D images so that the topological computation can be carried only within 2D in order to achieve computational efficiency. To enforce the continuity between slices, we propose a compound multi-slice representation and a compound multi-slice topological loss that incorporates rich topological information from adjacent slices. The quantitative and qualitative results show that our proposed method outperforms various strong baselines, especially for structure-related evaluation metrics.

Keywords: Machine learning, Inverse methods, Image registration
Abstract: Inverting a deformation field is a crucial part for numerous image registration methods and has an important impact on the final registration results. There are methods that work well for small and relatively simple deformations. However, a problem arises when the deformation field consists of complex and large deformations, potentially including folding. For such cases, the state-of-the-art methods fail and the inversion results are unpredictable. In this article, we propose a deep network using the encoder-decoder architecture to improve the inverse calculation. The network is trained using deformations randomly generated using various transformation models and their compositions, with a symmetric inverse consistency error as the cost function. The results are validated using synthetic deformations resembling real ones, as well as deformation fields calculated during registration of real histology data. We show that the proposed method provides an approximate inverse with a lower error than the current state-of-the-art methods.

Keywords: Histopathology imaging (e.g. whole slide imaging), Classification, Gastrointestinal tract
Abstract: Colorectal Cancer (CRC) is a leading cause of death around the globe, and therefore, the analysis of tumor micro environment in the CRC WSIs is important for the early detection of CRC. Conventional visual inspection is very time consuming and the process can undergo inaccuracies because of the subject-level assessment. Deep learning has shown promising results in medical image analysis. However, these approaches require a lot of labeling images from medical experts. In this paper, we propose a semi-supervised algorithm for CRC tissue classification. We propose to employ the hypergraph neural network to classify seven different biologically meaningful CRC tissue types. Firstly, image deep features are extracted from input patches using the pre-trained VGG19 model. The hypergraph is then constructed whereby patch-level deep features represent the vertices of hypergraph and hyperedges are assigned using pair-wise euclidean distance. The edges, vertices, and their corresponding patch-level features are passed through a feed-forward neural network to perform tissue classification in a transductive manner. Experiments are performed on an independent CRC tissue classification dataset and compared with existing state-of-the-art methods. Our results reveal that the proposed algorithm outperforms existing methods by achieving an overall accuracy of 95.46% and AvTP of 94.42%.

Keywords: X-ray imaging, Modeling - Image formation
Abstract: One of the most commonly used correction methods in X-ray imaging is flat field correction, which corrects for systematic inconsistencies, such as differences in detector pixel response. In conventional X-ray imaging, flat fields are acquired by exposing the detector without any object in the X-ray beam. However, in edge illumination X-ray CT, which is an emerging phase contrast imaging technique, two masks are used to measure the refraction of the X-rays. These masks remain in place while the flat fields are acquired and thus influence the intensity of the flat fields. This influence is studied theoretically and validated experimentally using Monte Carlo simulations of an edge illumination experiment in GATE.

Keywords: Image enhancement/restoration(noise and artifact reduction), Image segmentation, Microscopy - Light, Confocal, Fluorescence
Abstract: Precise quantification of the corneal nerve plexus morphology is of great importance in diagnosing peripheral diabetic neuropathy and assessing the progression of various eye-related systemic diseases, wherein segmentation of corneal nerves is an essential component. In this paper, we proposed and validated a novel pipeline for corneal nerve segmentation, comprising corneal confocal microscopy (CCM) image synthesis, image quality enhancement and nerve segmentation. Our goal was to address three major problems existing in most CCM datasets, namely inaccurate annotations, non-uniform illumination and contrast variations. In our synthesis and enhancement steps, we employed multilayer and patchwise contrastive learning based Generative Adversarial Network (GAN) frameworks, which took full advantage of multi-scale local features. Through both qualitative and quantitative experiments on two publicly available CCM datasets, our pipeline has achieved overwhelming enhancement performance compared to several state-of-the-art methods. Moreover, the segmentation results showed that models trained on our synthetic images performed much better than those trained on a real CCM dataset, which clearly identified the effectiveness of our synthesis method. Overall, our proposed pipeline can achieve satisfactory segmentation performance for poor-quality CCM images without using any manual labels and can effectively enhance those images.

Keywords: Image segmentation, Eye, Retinal imaging
Abstract: Retinal vessel segmentation is of great significance for clinical diagnosis of eye-related diseases and diabetic retinopathy. However, due to the imbalance of retinal vessel thickness distribution and the existence of a large number of capillaries, it is difficult to segment the retinal vessels correctly. To better solve this problem, we propose a novel Context Guided Attention Net (CGA-Net) with Joint hard sample mining strategy. Specifically, we propose a Context Guided Attention Module (CGAM) which can utilize both the surrounding context information and spatial attention information to promote the precision of segmentation results. As the CGAM is flexible and lightweight, it can be easily integrated into CNN architecture. To solve the problem of retinal vessel pixel imbalance, we further propose a novel Joint hard sample mining strategy (JHSM) in network training, which combines both the pixel-wise and patch-wise hard mining to largely improve the network’s robustness for hard samples. Experiments on publicly DRIVE and CHASE DB1 datasets show that our model outperforms state-of-the-art methods. Our code is available at https://github.com/chovy2019/medical_code.

Keywords: Machine learning, Kidney, Magnetic resonance imaging (MRI)
Abstract: Generalizing automated medical image segmentation methods to new image domains is inherently difficult. We have previously developed a number of automated segmentation methods that perform at the level of human readers on images acquired under similar conditions to the original training data. We are interested in exploring techniques that will improve model generalization to new imaging domains. In this study we explore a method to limit the inherent bias of these models to intensity and textural information. Using a dataset of 100 T2-weighted MR images with fat-saturation, and 100 T2-weighted MR images without fat-saturation, we explore the use of neural style transfer to induce shape preference and improve model performance on the task of segmenting the kidneys in patients affected by polycystic kidney disease. We find that using neural style transfer images improves the average dice value by ~0.2. In addition, visualizing individual network kernel responses highlights a drastic difference in the optimized networks. Biasing models to invoke shape preference is a promising approach to create methods that are more closely aligned with human perception.

Keywords: Image segmentation, Lung, Computed tomography (CT)
Abstract: Pulmonary lobe segmentation is an important preprocessing task for the analysis of lung diseases. Traditional methods relying on fissure detection or other anatomical features, such as the distribution of pulmonary vessels and airways, could provide reasonably accurate lobe segmentations. Deep learning based methods can outperform these traditional approaches, but require large datasets. Deep multi-task learning is expected to utilize labels of multiple different structures. However, commonly such labels are distributed over multiple datasets. In this paper, we proposed a multi-task semi-supervised model that can leverage information of multiple structures from unannotated datasets and datasets annotated with different structures. A focused alternating training strategy is presented to balance the different tasks. We evaluated the trained model on an external independent CT dataset. The results show that our model significantly outperforms single-task alternatives, improving the mean surface distance from 7.174 mm to 4.196 mm. We also demonstrated that our approach is successful for different network architectures as backbones.

Keywords: Ultrasound, Perfusion imaging, Quantification and estimation
Abstract: Contrast-enhanced ultrasound (CEUS) is a sensitive imaging technique to evaluate blood perfusion and tissue vascularity, whose quantification can assist in characterizing different perfusion patterns, e.g. in cancer or in arthritis. The perfusion parameters are estimated by fitting non-linear parametric models to experimental data, usually through the optimization of non-linear least squares, maximum likelihood, free energy or other methods that evaluate the adherence of a model adherence to the data. However, low signal-to-noise ratio and the nonlinearity of the model make the parameter estimation difficult. We investigate the possibility of providing estimates for the model parameters by directly analyzing the available data, without any fitting procedure, by using a deep convolutional neural network (CNN) that is trained on simulated ultrasound datasets of the model to be used. We demonstrated the feasibility of the proposed method both on simulated data and experimental CEUS data. In the simulations, the trained deep CNN performs better than constrained non-linear least squares in terms of accuracy of the parameter estimates, and is equivalent in term of sum of squared residuals (goodness of fit to the data). In the experimental CEUS data, the deep CNN trained on simulated data performs better than non-linear least squares in term of sum of squared residuals.

Keywords: Machine learning, fMRI analysis, Brain
Abstract: Traditional regression models do not generalize well when learning from small and noisy datasets. Here we propose a novel metamodel structure to improve the regression result. The metamodel is composed of multiple classification base models and a regression model built upon the base models. We test this structure on the prediction of autism spectrum disorder (ASD) severity as measured by the ADOS communication (ADOS_COMM) score from resting-state fMRI data, using a variety of base models. The metamodel outperforms traditional regression models as measured by the Pearson correlation coefficient between true and predicted scores and stability. In addition, we found that the metamodel is more flexible and more generalizable.

Keywords: Ultrasound, Inverse methods, Breast
Abstract: Ultrasound Computed Tomography (USCT) is an imaging method to map acoustic properties in soft tissues, e.g., for the diagnosis of breast cancer. A group of USCT methods rely on a passive reflector behind the imaged tissue, and they function by delineating such reflector in echo traces, e.g., to infer time-of-flight measurements for reconstructing local speed-of-sound maps. In this work, we study various echo features and delineation methods to robustly identify reflector profiles in echos. We compared and evaluated the methods on a multi-static data set of a realistic breast phantom. Based on our results, a RANSAC based outlier removal followed by an active contours based delineation using a new ``edge'' feature we propose that detects the first arrival times of echo performs robustly even in complex media; in particular 2.1 times superior to alternative approaches at locations where diffraction effects are prominent.

Keywords: Tooth, X-ray imaging, Image segmentation
Abstract: Accurate tooth segmentation in panoramic X-ray images is an essential stage before clinical surgery. This paper presents a deep segmentation network ToothPix, which leverages Generative Adversarial Network structures to exploit comprehensive semantic information for tooth segmentation. We introduce wide residual blocks and an encoder-decoder structure into the generator of ToothPix, which can learn grayscale and boundary features of teeth guided by a fully convolutional network discriminator. Without fine-grained ground truths, the losses in ToothPix guide the extraction of features to confuse the discriminator while effectively avoiding network overfitting. Furthermore, ToothPix generates small patches from whole panoramic X-ray images by a combination of image transformations, to increase the diversity of samples and reduce the computation. Experimental results demonstrate that our method outperforms state-of-the-art methods on the LNDb dental dataset.

Keywords: Magnetic resonance imaging (MRI), Machine learning, Image reconstruction - analytical & iterative methods
Abstract: Quantitative Susceptibility Mapping (QSM) is a new phase-based technique for quantifying magnetic susceptibility. The existing QSM reconstruction methods generally require complicated pre-processing on high-quality phase data. In this work, we propose to explore a new value of the high-pass filtered phase data generated in susceptibility weighted imaging (SWI), and develop an end-to-end Cross-connected Ψ-Net (CΨ-Net) to reconstruct QSM directly from these phase data in SWI without additional pre-processing. CΨ-Net adds an intermediate branch in the classical U-Net to form a Ψ-like structure. The specially designed dilated interaction block is embedded in each level of this branch to enlarge the receptive fields for capturing more susceptibility information from a wider spatial range of phase images. Moreover, the crossed connections are utilized between branches to implement a multi-resolution feature fusion scheme, which helps CΨ-Net capture rich contextual information for accurate reconstruction. The experimental results on a human dataset show that CΨ-Net achieves superior performance in our task over other QSM reconstruction algorithms.

Keywords: Shape analysis, Brain, Classification
Abstract: Geometric deep learning can find representations that are optimal for a given task and therefore improve the performance over pre-defined representations. While current work has mainly focused on point representations, meshes also contain connectivity information and are therefore a more comprehensive characterization of the underlying anatomical surface. In this work, we evaluate four recent geometric deep learning approaches that operate on mesh representations. These approaches can be grouped into template-free and template-based approaches, where the template-based methods need a more elaborate pre-processing step with the definition of a common reference template and correspondences. We compare the different networks for the prediction of Alzheimer’s disease based on the meshes of the hippocampus. Our results show advantages for template-based methods in terms of accuracy, number of learnable parameters, and training speed. While the template creation may be limiting for some applications, neuroimaging has a long history of building templates with automated tools readily available. Overall, working with meshes is more involved than working with simplistic point clouds, but they also offer new avenues for designing geometric deep learning architectures.

Keywords: Image segmentation, Radiation therapy, planing and treatment, Computed tomography (CT)
Abstract: Radiotherapy planning of head and neck cancer patients requires an accurate delineation of several organs at risk (OAR) from planning CT images in order to determine a dose plan which reduces toxicity and salvages normal tissue. However training a single deep neural network for multiple organs is highly sensitive to class imbalance and variability in size between several structures within the head and neck region. In this paper, we propose a single-class segmentation model for each OAR in order to handle class imbalance issues during training across output classes (one class per structure), where there exists a severe disparity between 12 OAR. Based on a U-net architecture, we present a transfer learning approach between similar OAR to leverage common learned features, as well as a simple weight averaging strategy to initialize a model as the average of multiple models, each trained on a separate organ. Experiments performed on an internal dataset of 200 H&N cancer patients treated with external beam radiotherapy, show the proposed model presents a significant improvement compared to the baseline multi-organ segmentation model, which attempts to simultaneously train several OAR. The proposed model yields an overall Dice score of 0.75 +/-0.12, by using both transfer learning across OAR and a weight averaging strategy, indicating that a reasonable segmentation performance can be achieved by leveraging additional data from surrounding structures, limiting the uncertainty in ground-truth annotations.

Keywords: Image enhancement/restoration(noise and artifact reduction), Animal models and imaging, Nuclear imaging (e.g. PET, SPECT)
Abstract: Missing areas in PET sinograms and severe image artifacts as a consequence thereof, still gain prominence not only in sparse-ring detector configurations but also in full-ring PET scanners in case of faulty detectors. Empty bins in the projection domain, caused by inter-block gap regions or any failure in the detector blocks may lead to unacceptable image distortions and inaccuracies in quantitative analysis. Deep neural networks have recently attracted enormous attention within the imaging community and are being deployed for various applications, including handling impaired sinograms and removing the streaking artifacts generated by incomplete projection views. Despite the promising results in sparse-view CT reconstruction, the utility of deep-learning-based methods in synthesizing artifact-free PET images in the sparse-crystal setting is poorly explored. Herein, we investigated the feasibility of a modified U-Net to generate artifact-free PET scans in the presence of severe dead regions between adjacent detector blocks on a dedicated high-resolution preclinical PET scanner. The performance of the model was assessed in both projection and image-space. The visual inspection and quantitative analysis seem to indicate that the proposed method is well suited for application on partial-ring PET scanners.

Keywords: Diffusion weighted imaging, Brain, Image segmentation
Abstract: We introduce the first hippocampus segmentation method for a novel high resolution (1×1×1mm³) diffusion tensor imaging (DTI) protocol acquired in 5.5 minutes at 3T. A new augmentation technique uses subsets of the DTI dataset to create mean diffusion weighted images (DWI) with plausible noise and contrast variations. The augmented DWI along with fractional anisotropy (FA) and mean diffusivity (MD) maps are used as inputs to a powerful convolutional neural network architecture. The method is evaluated for robustness using a second diffusion protocol.

Keywords: Machine learning, Microscopy - Light, Confocal, Fluorescence, Brain
Abstract: A fundamental challenge in machine learning-based segmentation of large-scale brain microscopy images is the time and domain expertise required by humans to generate ground truth for model training. Weakly supervised and semi-supervised approaches can greatly reduce the burden of human annotation. Here we present a study of three-dimensional U-Nets with varying levels of supervision to perform neuronal nuclei segmentation in light-sheet microscopy volumes. We leverage automated blob detection with classical algorithms to generate noisy labels on a large volume, and our experiments show that weak supervision, with or without additional fine-tuning, can outperform resource-limited fully supervised learning. These methods are extended to analyze coincidence between multiple fluorescent stains in cleared brain tissue. This is an initial step towards automated whole-brain analysis of plasticity-related gene expression.

Keywords: Computer-aided detection and diagnosis (CAD), Kidney, Computed tomography (CT)
Abstract: Renal cell carcinoma (RCC) is the most common and aggressive renal cancer. Hence, early identification of RCC is essential to provide the proper management plan. In this paper, we develop a novel computer-aided diagnostic (CAD) system that integrates texture and functional features, extracted from contrast-enhanced computed tomography (CE-CT), to differentiate benign from malignant RCC renal tumors and identify malignancy sub-types. Our study includes renal tumors obtained from 105 biopsy-proven cases of which 70 were diagnosed as malignant tumors (clear cell RCC (ccRCC) = 40 and non-clear cell RCC (nccRCC) = 30) and 35 were diagnosed as benign tumors (angiomyolipoma (AML) = 35). The proposed CAD system mainly consists of three steps: (i) preprocessing of grey images to obtain 3D segmented renal tumor objects; (ii) extracting different discriminating features (texture and functional) from segmented objects; and (ii) performing a two-stage classification process using different machine learning classifiers to obtain the final diagnosis of the renal tumor. In the first stage, the classification performance of the proposed CAD system was evaluated using the individual features along with a random forest machine learning classifier. Then, a weighted majority voting criteria was applied on the output class-membership to determine if the renal tumor is benign (AML) or malignant (RCC). In case of the latter, the second stage defines the sub-type of malignant tumor as ccRCC vs. nccRCC. Using a leave-one-subject-out cross-validation approach, the developed CAD system achieved 98.8% accuracy, 100% sensitivity, 89% specificity, and 0.97 F1 score in the first classification stage and achieved 71.4% accuracy and 0.75 F1 score in the second classification stage, respectively. These obtained results suggests that integrating first and second order texture features with functional features enhances the diagnostic performance of the developed CAD system making the developed a reliable noninvasive diagnostic tool for renal tumors.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Brain
Abstract: Radiomic features proposes a non invasive method for disease profiling. In the case of brain tumour studies, the quality of these features depends on the quality of tumour segmentation. However, these segmentations are not available for most cohorts. One way to address this issue is using object detection frameworks to automatically extract the area where the tumour is located in. The purpose of this study is to compare the quality of bounding-boxes based radiomics with manual segmentation, with regards to their performance in patient stratification and survival prediction.

Keywords: Magnetic resonance imaging (MRI), Prostate, Machine learning
Abstract: Radiomic features (RFs) based on multiparametric MRI (mpMRI) seem promising biomarkers of prostate cancer (PCa), although working with multiple mpMRI sequences makes standardization and proving RFs clinical reliability more challenging. Our study aims at investigating whether local RFs based on one-only high b-value Diffusion Weighted (DW) sequence can stratify patients according to four classes with progressive PCa risk levels. 42 biopsy-proven patients were enrolled, including patients with negative biopsy and either negative (n=7) or positive (n=10) mpMRI, NCS-PCa (n=10), and CS-PCa (n=15). 84 RFs measuring local heterogeneity were extracted from DWI_b2000, ranked based on Kruskal-Wallis (p<0.001) and one-tail Wilcoxon rank-sum test (p≤0.05) for multi- and pair-wise comparisons. RFs stability was assessed as segmentations varied. The Spearman index (ρ_s) assessed the rank correlation between RFs and risk levels. One RF, CV_L-m, stratifies patients in 4 progressive classes with ρ_s=0.81, thus suggesting that a progressive local tissue heterogeneity can predict PCa prognosis.

Keywords: Whole-body, Probabilistic and statistical models & methods, Image segmentation
Abstract: This paper presents an improvement of the keypoint transfer method for the segmentation of 3D medical images. Our approach is based on 3D SURF keypoint extraction, instead of 3D SIFT in the original algorithm. This yields a significantly higher number of keypoints, which allows to use a local segmentation transfer approach. The resulting segmentation accuracy is significantly increased, and smaller organs can be segmented correctly. We also propose a keypoint selection step which provides a good balance between speed and accuracy. We illustrate the efficiency of our approach with comparisons against state of the art methods.

Keywords: Shape analysis, Heart, Modeling - Anatomical, physiological and pathological
Abstract: In cases of pressure or volume overload, probing cardiac function may be difficult because of the interactions between shape and deformations. In this work, we use the LDDMM framework and parallel transport to estimate and reorient deformations of the right ventricle. We then propose a normalization procedure for the amplitude of the deformation, and a second-order spline model to represent the full cardiac contraction. The method is applied to 3D meshes of the right ventricle extracted from echocardiographic sequences of 314 patients divided into three disease categories and a control group. We find significant differences between pathologies in the model parameters, revealing insights into the dynamics of each disease.

Keywords: Pattern recognition and classification, Machine learning, Eye
Abstract: The increasing use of eye tracking in modern cognitive and clinical psychology, neuroscience, and ophthalmology requires new methods of objective quantitative analysis of complex eye movement data. In the current work, topological data analysis (TDA) is used to extract a new type of features of eye movements to differentiate between two eye movements groups, obtained upon the presentation of two different stimuli images - a human face, shown straight and rotated for 180 degrees, which corresponds to the processing of the normal and unusual visual information respectively. Experimental evidence shows that the proposed topology-based features have more discriminative power over the generally accepted features of eye movements, allowing to separate provided different stimuli with good accuracy. Moreover, the concatenation of the topology-based and ROI fixation ratios features further improves the performance of the classification task, showing the complementariness of the proposed topological features to the existing ones. We believe that the new class of features is able to serve as a valuable addition to the eye movement data-based medical diagnosis of mental, neurological, and ophthalmological disorders and diseases.

Keywords: Ultrasound, Heart, Image segmentation
Abstract: The analysis and segmentation of pediatric echocardiography is the critical steps for early diagnosis and timely treatment of congenital heart disease. However, the existing segmentation algorithms have the issues of information loss and low utilization of detail information, which reduces automatic segmentation accuracy. To solve this, we propose a multi-scale wavelet network (MS-Net) for pediatric echocardiographic segmentation. The MS-Net includes two branches: wavelet Unet (W-Unet) and bidirectional feature fusion (BFF-Net). In MS-Net, the discrete wavelet transform (DWT) is used to replace the sampling operation to solve information loss. In the first branch, BFF-Net achieves the fusion of context and detail information in low-resolution images by setting two top-down paths and a bottom-up path. In the second branch, W-Unet focuses on the extraction of high-resolution image details by reducing the network depth and propagation method. The information processing of the two branches is realized by feature fusion to solve the low utilization of detail information. The subjective and objective analysis on our self-collected pediatric echocardiographic dataset verify that the proposed algorithm achieves better segmentation accuracy than other commonly used algorithms.

Keywords: Computer-aided detection and diagnosis (CAD), Ultrasound, Machine learning
Abstract: We propose using a pre-trained segmentation model to perform diagnostic classification in order to achieve better generalization and interpretability, terming the technique reverse-transfer learning. We present an architecture to convert segmentation models to classification models. We compare and contrast dense vs sparse segmentation labeling and study its impact on diagnostic classification. We compare the performance of U-Net trained with dense and sparse labels to segment A-lines, B-lines, and Pleural lines on a custom dataset of lung ultrasound scans from 4 patients. Our experiments show that dense labels help reduce false positive detection. We study the classification capability of the dense and sparse trained U-Net and contrast it with a non-pretrained U-Net, to detect and differentiate COVID-19 and Pneumonia on a large ultrasound dataset of about 40k curvilinear and linear probe images. Our segmentation-based models perform better classification when using pretrained segmentation weights, with the dense-label pretrained U-Net performing the best.

Keywords: Image segmentation, Magnetic resonance imaging (MRI), Heart
Abstract: Pulmonary Hypertension (PH) induces anatomical changes in the cardiac muscle that can be quantitativly assessed using Magnetic Resonance (MR). Yet, the extraction of biomarkers relies on the segmentation of the affected structures, which in many cases is performed manually by physicians. Previous approaches have shown successful automatic segmentation results for different heart structures from human cardiac MR images. Nevertheless, the segmentation from mice images is rarely addressed, but it is essential for preclinical studies. Thus, the aim of this work is to develop an automatic tool based on a convolutional neural network for the segmentation of 4 cardiac structures at once in healthy and pathological mice to precisely evaluate biomarkers that may correlate to PH. The obtained automatic segmentations are comparable to manual segmentations, and they improve the distinction between control and pathological cases, especially regarding biomarkers from the right ventricle.

Keywords: Vessels, Machine learning, Magnetic resonance imaging (MRI)
Abstract: Intracranial vessel wall segmentation is critical for the quantitative assessment of intracranial atherosclerosis based on magnetic resonance vessel wall imaging. This work further improves on a previous 2D deep learning segmentation network by the utilization of 1) a 2.5D structure to balance network complexity and regularizing geometry continuity; 2) a UNET++ model to achieve structure adaptation; 3) an additional approximated Hausdorff distance (HD) loss into the objective to enhance geometry conformality; and 4) landing in a commonly used morphological measure of plaque burden - the normalized wall index (NWI) - to match the clinical endpoint. The modified network achieved Dice similarity coefficient of 0.9172 ± 0.0598 and 0.7833 ± 0.0867, HD of 0.3252 ± 0.5071 mm and 0.4914 ± 0.5743 mm, mean surface distance of 0.0940 ± 0.0781 mm and 0.1408 ± 0.0917 mm for the lumen and vessel wall, respectively. These results compare favorably to those obtained by the original 2D UNET on all segmentation metrics. Additionally, the proposed segmentation network reduced the mean absolute error in NWI from 0.0732 ± 0.0294 to 0.0725 ± 0.0333.

Keywords: Nuclear imaging (e.g. PET, SPECT), Motion compensation and analysis, Heart
Abstract: Cardiac SPECT perfusion imaging is important for diagnosis and evaluation of coronary artery diseases. However, the acquired image data are subject to motion blur due to patient respiratory motion. We propose a maximum-likelihood estimation (MLE) approach to determine a surrogate respiratory signal from short-time acquisition frames for motion correction. In the experiments we validated this approach first on a set of simulated phantom data with known respiratory motion, and then on clinical acquisitions from seven subjects. The results demonstrate that the proposed MLE approach could yield an accurate motion signal even with acquisition frame duration as short as 100 ms, and outperformed center-of-mass (CoM) and Laplacian eigenmaps (LE) methods.

Keywords: Machine learning, High-content (high-throughput) screening, Pattern recognition and classification
Abstract: Droplet-based microfluidic platforms arouse an increasing attention in various biomedical research by providing the isolated micro environment for biochemical reactions. Accordingly, it is of great significance to monitor and control the amount of the contents, e.g. cells or microbeads, inside each droplet. In this paper, we develop a novel weakly supervised algorithm to recognize droplets encapsulating diverse amount of cells (N-cell droplet encapsulation) from highly adherent droplet images. Quantitative experimental results exhibit that our approach can not only distinguish N-cell droplet encapsulations, but also locate each cell without any supervised location information.

Keywords: Multi-modality fusion, Genes, Histopathology imaging (e.g. whole slide imaging)
Abstract: Effective understanding of a disease such as cancer requires fusing multiple sources of information captured across physical scales by multimodal data. In this work, we propose a novel feature embedding module that derives from canonical correlation analyses to account for intra-modality and inter-modality correlations. Experiments on simulated and real data demonstrate how our proposed module can learn well-correlated multi-dimensional embeddings. These embeddings perform competitively on one-year survival classification of TCGA-BRCA breast cancer patients, yielding average F1 scores up to 58.69% under 5-fold cross-validation.

Keywords: Microscopy - Light, Confocal, Fluorescence, Brain, Probabilistic and statistical models & methods
Abstract: Light sheet microscopy of fluorescently labelled and optically cleared intact organs is a novel approach which enables to visualize structure and function of the brain in 3D. The methodology is gaining popularity in the neuroscience community and dedicated algorithms are being developed for segmenting and quantifying different neurological markers. However, comparisons of marker characteristics between the study groups are conventionally performed by conducting statistical testing for every atlas-defined brain region. While this statistical approach yields viable results, it is biased by the region delineations and does not provide information on the signal properties at a voxel level. In this work, we convert the 3D histological signal from segmented c-Fos+ cells into a format suitable for conducting voxel-wise group comparisons and demonstrate the potential of a recent technique, probabilistic threshold-free cluster enhancement method, in a brief comparative study of six different approaches to voxel-based statistical analysis.

Keywords: Histopathology imaging (e.g. whole slide imaging), Bone, Computer-aided detection and diagnosis (CAD)
Abstract: Automated analysis of tissue sections allows a better understanding of disease biology and may reveal biomarkers that could guide prognosis or treatment selection. In digital pathology, less abundant cell types can be of biological significance, but their scarcity can result in biased and sub-optimal cell detection model. To minimize the effect of cell imbalance on cell detection, we proposed a deep learning pipeline that considers the abundance of cell types during model training. Cell weight images were generated, which assign larger weights to less abundant cells and used the weights to regularize Dice overlap loss function. The model was trained and evaluated on myeloma bone marrow trephine samples. Our model obtained cell detection F1-score of 0.78, a 2% increase compared to baseline models, and it outperformed baseline models at detecting rare cell types. We found that scaling deep learning loss function by the abundance of cells improves cell detection performance. Our results demonstrate the importance of incorporating domain knowledge on deep learning methods for pathological data with class imbalance.

Keywords: Image reconstruction - analytical & iterative methods, Magnetic resonance imaging (MRI), Machine learning
Abstract: Deep neural networks have recently been thoroughly investigated as a powerful tool for MRI reconstruction. There is a lack of research, however, regarding their use for a specific setting of MRI, namely non-Cartesian acquisitions. In this work, we introduce a novel kind of deep neural networks to tackle this problem, namely density compensated unrolled neural networks, which rely on Density Compensation to correct the uneven weighting of the k-space. We assess their efficiency on the publicly available fastMRI dataset, and perform a small ablation study. Our results show that the density-compensated unrolled neural networks outperform the different baselines, and that all parts of the design are needed. We also open source our code, in particular a Non-Uniform Fast Fourier transform for TensorFlow.

Keywords: Magnetic resonance imaging (MRI), Image synthesis, Muscle
Abstract: Tagged magnetic resonance imaging (MRI) is a widely used imaging technique for measuring tissue deformation in moving organs. Due to tagged MRI's intrinsic low anatomical resolution, another matching set of cine MRI with higher resolution is sometimes acquired in the same scanning session to facilitate tissue segmentation, thus adding extra time and cost. To mitigate this, in this work, we propose a novel dual-cycle constrained bijective VAE-GAN approach to carry out tagged-to-cine MR image synthesis. Our method is based on a variational autoencoder backbone with cycle reconstruction constrained adversarial training to yield accurate and realistic cine MR images given tagged MR images. Our framework has been trained, validated, and tested using 1,768, 416, and 1,560 subject-independent paired slices of tagged and cine MRI from twenty healthy subjects, respectively, demonstrating superior performance over the comparison methods. Our method can potentially be used to reduce the extra acquisition time and cost, while maintaining the same workflow for further motion analyses.

Keywords: Optical coherence tomography, Eye, Image segmentation
Abstract: As a non-invasive imaging modality, Optical coherence tomography (OCT) has been widely used in clinical applications, mainly for monitoring the development of ophthalmic diseases. OCT can provide high-resolution images to reveal changes in retinal tissues, such as the accumulation of fluid in the retina caused by macular edema. This paper proposed an uncertainty-aware semi-supervised framework for retinal fluid segmentation. This framework composed of a teacher network and a student network, and they share the same network architecture. Each network consists of an encoder and three decoders. The three decoders in these two networks can simultaneously predict the probability map, contour map and distance map. A fusion operation is performed on the three maps predicted by the student network to obtain the segmentation results of the input images. The method proposed has been verified performance in the RETOUCH challenge dataset. The experimental results show the effectiveness of this method.

Keywords: Microscopy - Light, Confocal, Fluorescence, Image registration, In-vivo cellular and molecular imaging
Abstract: Non-rigid registration of cell nuclei in time-lapse microscopy images requires the estimation of nucleus deformation. We propose a new approach for deformation estimation and non-rigid registration of cell nuclei, which integrates elasticity constraints into a global optical flow-based method. We derive an elasticity prior on the deformation from the Navier equation. The common Markov random field prior in previous global methods is replaced by the elasticity prior to better regularize the estimated deformation fields. In addition, we introduce a scheme to exclude sub-cellular structures from estimating the nucleus deformation so that their relative motion does not deteriorate the registration result. Experiments on live cell microscopy image data demonstrate that the proposed method with elasticity constraints outperforms previous methods.

Keywords: Machine learning, Heart, Ultrasound
Abstract: There is considerable interest in AI systems that can assist a cardiologist to diagnose echocardiograms, and can also be used to train residents in classifying echocardiograms. Prior work has focused on the analysis of a single frame. Classifying echocardiograms at the video-level is challenging due to intra-frame and inter-frame noise. We propose a two-stream deep network which learns from the spatial context and optical flow for the classification of echocardiography videos. Each stream contains two parts: a Convolutional Neural Network (CNN) for spatial features and a bi-directional Long Short-Term Memory (LSTM) network with Attention for temporal. The features from these two streams are fused for classification. We verify our experimental results on a dataset of 170 (80 normal and 90 abnormal) videos that have been manually labeled by trained cardiologists. Our method provides an overall accuracy of 91.18%, with a sensitivity of 94.11% and a specificity of 88.24%.

Keywords: In-vivo cellular and molecular imaging, Image segmentation
Abstract: Recent advances in machine learning have shown significant success in biomedical image segmentation. Most existing high-quality segmentation algorithms rely on supervised learning with full training labels. However, such methods are more susceptible to label quality; besides, generating accurate labels in biomedical data is a labor- and time-intensive task. In this paper, we propose a novel neuron segmentation method that uses only incomplete and noisy labels. The proposed method employs a noise-tolerant adaptive loss that handles partially annotated labels. Moreover, the proposed reconstruction loss leverages prior knowledge of neuronal cell structures to reduce false segmentation near noisy labels. the proposed loss function outperforms several widely used state-of-the-art noise-tolerant losses, such as reverse cross entropy, normalized cross entropy and noise-robust dice losses.

Keywords: Image segmentation, Probabilistic and statistical models & methods, Endoscopy
Abstract: For medical image segmentation tasks, some of foreground objects have more ambiguities than other areas because of confusing appearances. It is critical to seek a proper method to measure such ambiguity of each pixel and use it for robust model training. To this end, we design a Bayesian uncertainty estimate layer, and propose an uncertainty-guided training for standard convolutional segmentation models. In particular, the proposed Bayesian uncertainty estimate layer provides the confidence on each pixel’s prediction independently, and works with prediction correctness to obtain the rescaling weights of training loss for each pixel. Through this mechanism, the learning importance of the regions with different ambiguities can be distinguished. We validate our proposal by comparing it with other loss rescaling approaches on medical image datasets. The results consistently show that the uncertainty-guided training brings significant improvement on lesion segmentation accuracy.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Brain
Abstract: Overall survival (OS) time prediction is one of the most common estimates of the prognosis of gliomas and is used to design an appropriate treatment planning. State-of-the-art (SOTA) methods for OS time prediction follow a pre-hoc approach that require computing the segmentation map of the glioma tumor sub-regions (necrotic, edema tumor, enhancing tumor) for estimating OS time. However, the training of the segmentation methods require ground truth segmentation labels which are tedious and expensive to obtain. Given that most of the large-scale data sets available from hospitals are unlikely to contain such precise segmentation, those SOTA methods have limited applicability. In this paper, we introduce a new post-hoc method for OS time prediction that does not require segmentation map annotation for training. Our model uses medical image and patient demographics (represented by age) as inputs to estimate the OS time and to estimate a saliency map that localizes the tumor as a way to explain the OS time prediction in a post-hoc manner. It is worth emphasizing that although our model can localize tumors, it uses only the ground truth OS time as training signal, i.e., no segmentation labels are needed. We evaluate our post-hoc method on the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2019 data set and show that it achieves competitive results compared to pre-hoc methods with the advantage of not requiring segmentation labels for training. We make our code available at https://github.com/renato145/posthocOS.

Keywords: Machine learning, Computer-aided detection and diagnosis (CAD), Classification
Abstract: Traditionally, convolutional neural networks need large amounts of data labelled by humans to train. Self supervision has been proposed as a method of dealing with small amounts of labelled data. The aim of this study is to determine whether self supervision can increase classification performance on a small COVID-19 CT scan dataset. This study also aims to determine whether the proposed self supervision strategy, targeted self supervision, is a viable option for a COVID-19 imaging dataset. A total of 10 experiments are run comparing the classification performance of the proposed method of self supervision with different amounts of data. The experiments run with the proposed self supervision strategy perform significantly better than their non-self supervised counterparts. We get almost 6% increase on average with self supervision compared to no self supervision, and more than 8% increase in accuracy in our best run with self supervision when compared to no self supervision. The results suggest that self supervision can improve classification performance on a small COVID-19 CT scan dataset. Code for targeted self supervision can be found at this link: https://github.com/Mewtwo/Targeted-Self-Supervision/tree/main/COVID-CT

Keywords: Shape analysis, Magnetic resonance imaging (MRI), Brain
Abstract: The problem of inter-individual comparison is of major importance in neuroimaging to detect patterns indicative of neurological pathology. Few works have been addressing the comparison of individual sulcal graphs in which variations across subjects manifest as changes in the number of nodes, graph topology and in the attributes that can be attached to nodes and edges. Here, we quantitatively evaluated different graph matching approaches in both the pairwise and multi-graph matching frameworks, on synthetic graphs simulating the structure and attributes distributions of real data. Our results show that multigraph matching approach outperforms pairwise techniques in all simulations. The application to a set of real sulcal graphs from 134 subjects confirms this observation and demonstrates that multigraph matching approaches can scale and have a great potential in this context.

Keywords: Computer-aided detection and diagnosis (CAD), Machine learning, Image segmentation
Abstract: Performing coarse-to-fine abdominal multi-organ segmentation facilitates extraction of high-resolution segmentation minimizing the loss of spatial contextual information. However, current coarse-to-refine approaches require a significant number of models to perform single organ segmentation. We propose a coarse-to-fine pipeline RAP-Net, which starts from the extraction of the global prior context of multiple organs from 3D volumes using a low-resolution coarse network, followed by a fine phase that uses a single refined model to segment all abdominal organs instead of multiple organ corresponding models. We combine the anatomical prior with corresponding extracted patches to preserve the anatomical locations and boundary information for performing high-resolution segmentation across all organs in a single model. To train and evaluate our method, a clinical research cohort consisting of 100 patient volumes with 13 organs well-annotated is used. We tested our algorithms with 4-fold cross-validation and computed the Dice score for evaluating the segmentation performance of the 13 organs. Our proposed method using single auto-context outperforms the state-of-the-art on 13 models with an average Dice score 84.58% versus 81.69% (p<0.0001).

Keywords: Image synthesis, Magnetic resonance imaging (MRI), Brain
Abstract: Changes in brain morphology, such as cortical thinning are of great value for understanding the trajectory of brain aging and various neurodegenerative diseases. In this work, we employed a generative neural network variational autoencoder (VAE) that is conditional on age and is able to generate cortical thickness maps at various ages given an input cortical thickness map. To take into account the mesh topology in the model, we proposed a loss function based on weighted adjacency to integrate the surface topography defined as edge connections with the cortical thickness mapped as vertices. Compared to traditional conditional VAE that did not use the surface topological information, our method better predicted “future” cortical thickness maps, especially when the age gap became wider. Our model has the potential to predict the distinctive temporospatial pattern of individual cortical morphology in relation to aging and neurodegenerative diseases.

Keywords: Computed tomography (CT), Inner ear, Image enhancement/restoration(noise and artifact reduction)
Abstract: Existing medical image super-resolution methods rely on pairs of low- and high- resolution images to learn a mapping in a fully supervised manner. However, such image pairs are often not available in clinical practice. In this paper, we address the super-resolution problem in a real-world scenario using unpaired data and synthesize linearly eight times higher resolved Micro-CT images of temporal bone structure, which is embedded in the inner ear. We explore cycle-consistency generative adversarial networks for super-resolution task and equip the translation approach with Bayesian inference. We further introduce Hu Moment the evaluation metric to quantify the structure of the temporal bone. We evaluate our method on a public inner ear CT dataset and have seen both visual and quantitative improvement over state-of-the-art deep-learning-based methods. In addition, we perform a multi-rater visual evaluation experiment and find that trained experts consistently rate the proposed method highest quality scores among all methods. Implementing our approach as an end-to-end learning task, we are able to quantify uncertainty in the unpaired translation tasks and find that the uncertainty mask can provide structural information of the temporal bone.