Keywords: Histopathology imaging (e.g. whole slide imaging), Integration of multiscale information, Atlases
Abstract: Tau protein neurofibrillary tangles (NFT) are linked to neuronal and synaptic loss and cognitive decline in Alzheimer's disease (AD) and related dementias. In AD, NFT pathology is known to spread through the cortex in a characteristic pattern, starting in the medial temporal lobe. However, the exact 3D pattern of NFT progression has not been described, and capturing this pattern quantitatively can help inform in vivo AD imaging biomarkers. We present a computational framework for generating 3D maps of NFT load from ex vivo MRI and serial histology. Weakly supervised deep learning is used to detect NFTs on histology slides prepared with an anti-tau immunohistochemistry stain, and a multi-stage registration pipeline that leverages 3D printing is used for histology-MRI alignment. Derived maps of NFT density are strongly concordant with manual NFT counting, as well as categorical NFT severity ratings used for clinical diagnosis.

Keywords: Machine learning, Brain, Magnetic resonance imaging (MRI)
Abstract: There is an increasing need for efficient and automatic evaluation of brain tumors on magnetic resonance images (MRI). Most of the previous works focus on segmentation, registration, and growth modeling of the most common primary brain tumor gliomas, or the classification of up to three types of brain tumors. In this work, we extend the study to eight types of brain tumors where only global diagnosis labels are given but not the slice-level labels. We propose a weakly supervised method and demonstrate that inferring disease types at the slice-level would help the global label prediction. We also provide an algorithm for feature extraction via randomly choosing connection paths through class-specific autoencoders with dropout to accommodate the small-dataset problem. Experimental results on both public and proprietary datasets are compared to the baseline methods. The classification with the weakly supervised setting on the proprietary data, consisting of 295 patients with eight different tumor types, shows close results to the upper bound in the supervised learning setting.

Keywords: Machine learning, Brain, Magnetic resonance imaging (MRI)
Abstract: In neuroimaging studies, the human cortex is commonly mod elled as a sphere to preserve the topological structure of the cortical surface. In this work, we explore the analysis of the human cortex using spherical CNN in an Alzheimer’s disease (AD) classification task, using morphometric measures derived from T1-weighted structural MRI. Our results show superior performance in classifying AD versus cognitively normal subjects and in predicting mild cognitive impairment progression within two years. This work demonstrates the feasibility and superiority of the spherical CNN directly applied on the spherical representation in the discriminative analysis of the human cortex and could be readily extended to other imaging modalities and other neurological diseases.

Keywords: Machine learning, Brain, Magnetic resonance imaging (MRI)
Abstract: Predicting the future course of a disease with limited information is an essential but challenging problem in health care. For older adults, especially the ones suffering from Alzheimer's disease, accurate prediction of their longitudinal trajectories of cognitive decline can facilitate appropriate prognostic clinical action. Increasing evidence has shown that longitudinal brain imaging data can aid in the prediction of cognitive trajectories. However, in many cases, only a single (baseline) measurement from imaging is available for prediction. We propose a novel model for predicting the trajectory of cognition, using only a baseline measurement, by leveraging the temporal dependence in cognition. On both a synthetic dataset and a real-world dataset, we demonstrate that our model is superior to prior approaches in predicting cognition trajectory over the next five years. We show that the model's ability to capture nonlinear interaction between features leads to improved performance. Further, the proposed model achieved significantly improved trajectory prediction in subjects at higher risk of cognitive decline (those with genetic risk and worse clinical profiles at baseline), highlighting its clinical utility.

Keywords: fMRI analysis, Computer-aided detection and diagnosis (CAD)
Abstract: Functional connectivity (FC) analysis is an appealing tool to aid diagnosis and elucidate the neurophysiological underpinnings of autism spectrum disorder (ASD). Many machine learning methods have been developed to distinguish ASD patients from healthy controls based on FC measures and identify abnormal FC patterns of ASD. Particularly, several studies have demonstrated that deep learning models could achieve better performance for ASD diagnosis than conventional machine learning methods. Although promising classification performance has been achieved by the existing machine learning methods, they do not explicitly model heterogeneity of ASD, incapable of disentangling heterogeneous FC patterns of ASD. To achieve an improved diagnosis and a better understanding of ASD, we adopt capsule networks (CapsNets) to build classifiers for distinguishing ASD patients from healthy controls based on FC measures and stratify ASD patients into groups with distinct FC patterns. Evaluation results based on a large multi-site dataset have demonstrated that our method not only obtained better classification performance than state-of-the-art alternative machine learning methods, but also identified clinically meaningful subgroups of ASD patients based on their vectorized classification outputs of the CapsNets classification model.

Keywords: Machine learning, Modeling - Knowledge, Brain
Abstract: We investigated the ability of graph convolutional network (GCN) that takes into account the mesh topology as a sparse graph to predict brain age for preterm neonates using cortical surface morphometrics, i.e. cortical thickness and sulcal depth. Compared to machine learning and deep learning methods that did not use the surface topological information, the GCN better predicted the ages for preterm neonates with none/mild perinatal brain injuries (NMI). We then tested the GCN trained using NMI brains to predict the age of neonates with severe brain injuries (SI). Results also displayed good accuracy (MAE=1.43 weeks), while the analysis of the interaction term (true age × group) showed that the slope of the predicted brain age relative to the true age for the SI group was significantly less steep than the NMI group (p<0.0001), indicating that SI can decelerate early postnatal growth. To understand regional contributions to age prediction, we applied GCNs separately to the vertices within each cortical parcellation. The middle cingulate cortex that is known to be one of the thickest cortical regions in the neonatal period showed the best accuracy in age prediction (MAE = 1.24 weeks). Furthermore, we found that the regional brain ages computed using GCN models in several frontal cortices significantly correlated with cognitive abilities at 3 years of age. Furthermore, the brain predicted age in part of the superior temporal cortex, which is the auditory and language processing locus, was related to language functional scores at 3 years. Our results demonstrate the potential of the GCN models for predicting brain age as well as localizing brain regions contributing to the prediction of age and future cognitive outcome.

Keywords: Computer-aided detection and diagnosis (CAD), Breast, Classification
Abstract: Meta-training has been empirically demonstrated to be the most effective pre-training method for few-shot learning of medical image classifiers (i.e., classifiers modeled with small training sets). However, the effectiveness of meta-training relies on the availability of a reasonable number of hand-designed classification tasks, which are costly to obtain, and consequently rarely available. In this paper, we propose a new method to unsupervisedly design a large number of classification tasks to meta-train medical image classifiers. We evaluate our method on a breast dynamically contrast enhanced magnetic resonance imaging (DCE-MRI) data set that has been used to benchmark few-shot training methods of medical image classifiers. Our results show that the proposed unsupervised task design to meta-train medical image classifiers builds a pre-trained model that, after fine-tuning, produces better classification results than other unsupervised and supervised pre-training methods, and competitive results with respect to meta-training that relies on hand-designed classification tasks.

Keywords: Prostate, Classification, Magnetic resonance imaging (MRI)
Abstract: The use of quantitative radiomic features of MRI to predict the aggressiveness of prostate cancer has attracted increasing amounts of attention due to its potential as a non-invasive biomarker for prostate cancer. In this study, to predict prostate cancer aggressiveness, we investigate the usefulness of multi-modal radiomic features according to combination method such as concatenation or averaging and compare multi-modal radiomic features to uni-modal radiomic features. To define the prostate cancer region of T2wMR based on ground truth pathology, a radiologist manually segmented prostate cancer referring to a fusion result of registration of histopathology image and T2wMR. The prostate cancer region of the ADC is then defined as the same region as the T2wMR through registration of the ADC on the T2wMR. To extract radiomic features to predict prostate cancer aggressiveness, total 68 features are calculated for each region of T2wMR and ADC. To predict the aggressiveness of prostate cancer, a random forest classifier is trained for each region in T2wMR and ADC. The prostate cancer regions were categorized as Low GS Group (GS <= 3+4) and High GS Group (GS >= 4+3). As results, the sensitivity of combined features was the highest at 82.0% which is higher 2.4%, 2.7% and 4.8% than ADC, T2wMR, and average combined features. Experiment results showed that the possibility of determining the aggressiveness of prostate cancer through the multi-modal radiomic features of T2wMR and ADC.

Keywords: Ultrasound, Kidney, Computer-aided detection and diagnosis (CAD)
Abstract: Ultrasound images are widely used for diagnosis of congenital abnormalities of the kidney and urinary tract (CAKUT). Since a typical clinical ultrasound image captures 2D information of a specific view plan of the kidney and images of the same kidney on different planes have varied appearances, it is challenging to develop a computer aided diagnosis tool robust to ultrasound images in different views. To overcome this problem, we develop a multi-instance deep learning method for distinguishing children with CAKUT from controls based on their clinical ultrasound images, aiming to automatic diagnose the CAKUT in children based on ultrasound imaging data. Particularly, a multi-instance deep learning method was developed to build a robust pattern classifier to distinguish children with CAKUT from controls based on their ultrasound images in sagittal and transverse views obtained during routine clinical care. The classifier was built on imaging features derived using transfer learning from a pre-trained deep learning model with a mean pooling operator for fusing instance-level classification results. Experimental results have demonstrated that the multi-instance deep learning classifier performed better than classifiers built on either individual sagittal slices or individual transverse slices.

Keywords: Ultrasound, Liver, Machine learning
Abstract: Liver vessels can be visualized at sub-millimetre scale with contrast-enhanced ultrasound. In this work we exploit a co- hort of 97 subjects (healthy volunteers and 4 liver disease types), exploiting multiple videos acquired at locations within the liver hand-picked by the sonographer to perform the diag- nostic task. Annotation was performed at subject-level (dis- ease subtype or healthy), along with scoring of image quality. We propose an original approach exploiting the abstraction capabilities of maximum intensity projections (MIPs) to feed a deep-learning classifier. Two architectures were tested for which we compared performance with different scenarios re- garding the exploitation of transfer learning and the number of input MIPs per subjects. Our results show over 88% accu- racy for a 2-class task (healthy versus disease), and 70% for a 3-class task (healthy versus 2 disease sub-types). We demon- strate, for the first time, that deep learning with minimal su- pervision and no pre-filtering can accurately classify liver dis- eases based on vascular ultrasound imaging acquired in a clin- ical setting. We also report findings on specific misclassica- tion patterns which will guide further studies, augmentation of the cohort and subject annotation.

Keywords: Computer-aided detection and diagnosis (CAD), Prostate, Magnetic resonance imaging (MRI)
Abstract: Prostate cancer (PCa) is the most prevalent and one of the leading causes of cancer death among men. Multi-parametric MRI (mp-MRI) is a prominent diagnostic scan, which could help in avoiding unnecessary biopsies for men screened for PCa. Artificial intelligence (AI) systems could help radiologists to be more accurate and consistent in diagnosing clinically significant cancer from mp-MRI scans. Lack of specificity has been identified recently as one of weak points of such assistance systems. In this paper, we propose a novel false positive reduction network to be added to the overall detection system to further analyze lesion candidates. The new network utilizes multiscale 2D image stacks of these candidates to discriminate between true and false positive detections. We trained and validated our network on a dataset with 2090 cases from seven different institutions and tested it on a separate independent dataset with 243 cases. With the proposed model, we achieved area under curve (AUC) of 0.876 on discriminating between true and false positive detected lesions and improved the AUC from 0.825 to 0.867 on overall identification of clinically significant cases.

Keywords: Computer-aided detection and diagnosis (CAD), Endoscopy, Gastrointestinal tract
Abstract: Computer-aided polyp detection during colonoscopy is beneficial to reduce the risk of colorectal cancers. Deep learning techniques have made significant process in natural object detection. However, when applying those fully supervised methods to polyp detection, the performance is greatly depressed by the deficiency of labeled data. In this paper, we propose a novel bootstrapping method for polyp detection in colonoscopy videos by augmenting training data with temporal consistency. For a detection network that is trained on a small set of annotated polyp images, we fine-tune it with new samples selected from the test video itself, in order to more effectively represent the polyp morphology of current video. A strategy of selecting new samples is proposed by considering temporal consistency in the test video. Evaluated on 11954 endoscopic frames of the CVC-ClinicVideoDB dataset, our method yields great improvement on polyp detection for several detection networks, and achieves state-of-the-art performance on the benchmark dataset.

Keywords: Vessels, Image segmentation, Machine learning
Abstract: Deep convolutional neural networks have proved effective in segmenting lesions and anatomies in various medical imaging modalities. However, in the presence of small sample size and domain shift problems, these models often produce masks with non-intuitive segmentation mistakes. In this paper, we propose a segmentation framework called ErrorNet, which learns to correct these segmentation mistakes through the repeated process of injecting systematic segmentation errors to the segmentation result based on a learned shape prior, followed by attempting to predict the injected error. During inference, ErrorNet corrects the segmentation mistakes by adding the predicted error map to the initial segmentation result. ErrorNet has advantages over alternatives based on domain adaptation or CRF-based post processing, because it requires neither domain-specific parameter tuning nor any data from the target domains. We have evaluated ErrorNet using five public datasets for the task of retinal vessel segmentation. The selected datasets differ in size and patient population, allowing us to evaluate the effectiveness of ErrorNet in handling small sample size and domain shift problems. Our experiments demonstrate that ErrorNet outperforms a base segmentation model, a CRF-based post processing scheme, and a domain adaptation method, with a greater performance gain in the presence of the aforementioned dataset limitations.

Keywords: Retinal imaging, Machine learning, Computer-aided detection and diagnosis (CAD)
Abstract: In this paper, we proposed and validated a novel and effective pipeline for automatically detecting hemorrhage from coarsely-annotated fundus images in diabetic retinopathy. The proposed framework consisted of three parts: image preprocessing, training data refining, and object detection using a convolutional neural network with label smoothing. Contrast limited adaptive histogram equalization and adaptive gamma correction with weighting distribution were adopted to improve image quality by enhancing image contrast and correcting image illumination. To refine coarsely-annotated training data, we designed a bounding box refining network (BBR-net) to provide more accurate bounding box annotations. Combined with label smoothing, RetinaNet was implemented to alleviate mislabeling issues and automatically detect hemorrhages. The proposed method was trained and evaluated on a publicly available IDRiD dataset and also one of our private datasets. Experimental results showed that our BBR-net could effectively refine manually-delineated coarse hemorrhage annotations, with the average IoU being 0.8715 when compared with well-annotated bounding boxes. The proposed hemorrhage detection pipeline was compared to several alternatives and superior performance was observed.

Keywords: Retinal imaging, Eye, Pattern recognition and classification
Abstract: Image classification using deep convolutional neural networks (DCNN) has a competitive performance as compared to other state-of-the-art methods. Here, attention can be visualized as a heatmap to improve the explainability of DCNN. We generated the initial heatmaps by using gradient-based classification activation map (Grad-CAM). We first assume that these Grad-CAM heatmaps can reveal the lesion regions well, then apply the attention mining on these heatmaps. Another, we assume that these Grad-CAM heatmaps can't reveal the lesion regions well then apply the dissimilarity loss on these Grad-CAM heatmaps. In this study, we asked the ophthalmologists to select 30% of the heatmaps. Furthermore, we design knowledge preservation (KP) loss to minimize the discrepancy between heatmaps generated from the updated network and the selected heatmaps. Experiments revealed that our method improved accuracy from 90.1% to 96.2%. We also found that the attention regions are closer to the GT lesion regions.

Keywords: Classification, Computer-aided detection and diagnosis (CAD), Retinal imaging
Abstract: Diabetic retinopathy (DR), as a leading ocular disease, is often with a complication of diabetic macular edema (DME). However, most existing works only aim at DR grading but ignore the DME diagnosis, but doctors will do both tasks simultaneously. In this paper, motivated by the advantages of multi-task learning for image classification, and to mimic the behavior of clinicians in visual inspection for patients, we propose a feature Separation and Union Network (SUNet) for simultaneous DR and DME grading. Further, to improve the interpretability of the disease grading, a lesion regularizer is also imposed to regularize our network. Specifically, given an image, our SUNet first extracts a common feature for both DR and DME grading and lesion detection. Then a feature blending block is introduced which alternately uses feature separation and feature union for task-specific feature extraction,where feature separation learns task-specific features for lesion detection and DR and DME grading, and feature union aggregates features corresponding to lesion detection, DR and DME grading. In this way, we can distill the irrelevant features and leverage features of different but related tasks to improve the performance of each given task. Then the taskspecific features of the same task at different feature separation steps are concatenated for the prediction of each task. Extensive experiments on the very challenging IDRiD dataset demonstrate that our SUNet significantly outperforms existing methods for both DR and DME grading.

Keywords: Machine learning, Retinal imaging, Eye
Abstract: Adaptive optics (AO) scanning laser ophthalmoscopy offers cellular level in-vivo imaging of the human cone mosaic. Existing analysis of cone photoreceptor density in AO images require accurate identification of cone cells, which is a time and labor-intensive task. Recently, several methods have been introduced for automated cone detection in AO retinal images using convolutional neural networks (CNN). However, these approaches have been limited in their ability to correctly identify cones when applied to AO images originating from different locations in the retina, due to changes to the reflectance and arrangement of the cone mosaics with eccentricity. To address these limitations, we present an adapted CNN architecture that incorporates spatial information directly into the network. Our approach, inspired by conditional generative adversarial networks, embeds the retina location from which each AO image was acquired as part of the training. Using manual cone identification as ground truth, our evaluation shows general improvement over existing approaches when detecting cones in the middle and periphery regions of the retina, but decreased performance near the fovea.

Keywords: Optical coherence tomography, Eye, Computer-aided detection and diagnosis (CAD)
Abstract: As one of the major types of glaucoma, closed-angle glaucoma is the leading cause of irreversible blindness in the world. The ability of Anterior Segment Optical Coherence Tomography (AS-OCT) to obtain high-resolution cross-sectional images of the entire anterior chamber in a single image makes it an important tool for glaucoma diagnosis. In this paper, we propose a practical and efficient system based on deep learning to accurately classify anterior chamber angle (ACA) closure by using the location of scleral spur (SS) points. First, the localization problem is reformulated as a pixel-wise regression task. A fully convolutional deep neural network is optimized to predict the probability that each pixel belongs to the SS points, and the numerical coordinates are obtained by the maximum likelihood estimation theory. Second, the ACA region centered on the detected SS is cropped as the input of the classification model. The single model applied for classification is SE-ResNet18 and optimized with focal loss. In the AGE Challenge 2019[1], our proposed method obtained superior performance for angle-closure glaucoma screening.

Keywords: fMRI analysis, Classification, Machine learning
Abstract: Twin study is one of the major parts of human brain research that reveals the importance of environmental and genetic influences on different aspects of brain behavior and disorders. Accurate characterization of identical and fraternal twins allows us to inference on the genetic influence in a population. In this paper, we propose a novel pairwise classification pipeline to identify the zygosity of twin pairs using the resting state functional magnetic resonance images (rs-fMRI). The new feature representation is utilized to efficiently construct brain network for each subject. Specifically, we project the fMRI signal to a set of cosine series basis and use the projection coefficients as the compact and discriminative feature representation of noisy fMRI. The pairwise relation is encoded by a set of twinwise correlations between functional brain networks across brain regions. We further employ hill climbing variable selection to identify the most genetically affected brain regions. The proposed framework has been applied to 208 twin pairs in Human Connectome Project (HCP) and we achieved 92.23(±4.43)% classification accuracy.

Keywords: fMRI analysis, Functional imaging (e.g. fMRI), Brain
Abstract: We propose a method for estimating more reproducible functional networks that are more strongly associated with dynamic task activity by using recurrent neural networks with long short term memory (LSTMs). The LSTM model is trained in an unsupervised manner to learn to generate the functional magnetic resonance imaging (fMRI) time-series data in regions of interest. The learned functional networks can then be used for further analysis, e.g., correlation analysis to determine functional networks that are strongly associated with an fMRI task paradigm. We test our approach and compare to other methods for decomposing functional networks from fMRI activity on 2 related but separate datasets that employ a biological motion perception task. We demonstrate that the functional networks learned by the LSTM model are more strongly associated with the task activity and dynamics compared to other approaches. Furthermore, the patterns of network association are more closely replicated across subjects within the same dataset as well as across datasets. More reproducible functional networks are essential for better characterizing the neural correlates of a target task.

Keywords: Machine learning, fMRI analysis, Data Mining
Abstract: Recent studies showed that convolutional neural network (CNN) models possess remarkable capability of differentiating and characterizing fMRI signals from cortical gyri and sulci. In addition, visualization and analysis of the filters in the learned CNN models suggest that sulcal fMRI signals are more diverse and have higher frequency than gyral signals. However, it is not clear whether the gyral fMRI signals can be further divided into sub-populations, e.g., 3-hinge areas vs 2-hinge areas. It is also unclear whether the CNN models of two classes (gyral vs sulcal) classification can be further optimized for three classes (3-hinge gyral vs 2-hinge gyral vs sulcal) classification. To answer these questions, in this paper, we employed the AdaNet framework to design a neural architecture search (NAS) system for optimizing CNN models for three classes fMRI signal classification. The core idea is that AdaNet adaptively learns both the optimal structure of the CNN network and its weights so that the learnt CNN model can effectively extract discriminative features that maximize the classification accuracies of three classes of 3-hinge gyral, 2-hinge gyral and sulcal fMRI signals. We evaluated our framework on the Autism Brain Imaging Data Exchange (ABIDE) dataset, and experiments showed that our framework can obtained significantly better results, in terms of both classification accuracy and extracted features.

Keywords: Functional imaging (e.g. fMRI), Brain, Computer-aided detection and diagnosis (CAD)
Abstract: Autism is a developmental disorder associated with difficulties in communication and social interaction. Currently, the gold standard in autism diagnosis is the autism diagnostic observation schedule (ADOS) interviews that assign a score indicating the level of severity for each individual. However, current researchers investigate developing objective technologies to diagnose autism employing brain image modalities. One of such image modalities is task-based functional MRI which exhibits alterations in functional activity that is believed to be important in explaining autism causative factors. Although autism is defined over a wide spectrum, previous diagnosis approaches only divide subjects into normal or autistic. In this paper, a novel framework for grading the severity level of autistic subjects using task-based fMRI data is presented. A speech experiment is used to obtain local features related to the functional activity of the brain. According to ADOS reports, the adopted dataset of 39 subjects is classified to three groups (13 subjects per group): mild, moderate and severe. Individual analysis with the general linear model (GLM) is used for feature extraction for each 246 brain areas according to the Brainnetome atlas (BNT). Our classification results are obtained by random forest classifier after recursive feature elimination (RFE) with 72% accuracy. Finally, we validate our selected features by applying higher-level group analysis to prove how informative they are and to infer the significant statistical differences between groups

Keywords: Image enhancement/restoration(noise and artifact reduction), EEG & MEG, Functional imaging (e.g. fMRI)
Abstract: EEG recorded during MRI acquisition suffers from severe artifacts due to the imaging gradients. Here, we explore the possibility of using denoising autoencoders for correcting for these artifacts. After hyperparameter optimization, the performance of the algorithm was compared against PCA on two different synthesized datasets. The first dataset was created by adding a template artifact to clean EEG data and randomly shifting it in time to simulate aliasing. While the second dataset was formed by filtering out the EEG frequencies and adding a known ground-truth clean EEG signal. The performance of each method was assessed by the RMSE relative to the clean EEG signal. In addition, the correlation coefficient compared to the artifact signal was used to measure the residual artifact level. On the second synthesized dataset, denoising autoencoders outperformed PCA by 4.3% in terms of RMSE, meaning they were able to better preserve the original signal while at the same time the correlation with the underlying artifact was reduced by 40%. These preliminary results merit further investigation on a larger dataset.

Keywords: Image registration, Functional imaging (e.g. fMRI), Brain
Abstract: A well-known challenge in fMRI data analysis is the excessive variability in the MR signal and the high level of random and structured noise. A common solution to deal with such high variability/noice is to recruit a large number of subjects to enhance the statistical power to detect any scientific effect. To achieve this, the morphologies of the sample brains are required to be warped into a standard space. However, human's cerebral cortices are highly convoluted, with large inter-subject morphological variability that renders the task of registration challenging. Currently, the state-of-the-art non-linear registration methods perform poorly on brains' cortical regions particularly on aging and clinical populations. To alleviate this issue, we propose a landmark-guided and region-based image registration method. We have evaluated our method by warping the brain cortical regions of both young and older participants into the standard space. Compared with the state-of-the-art method, we showed our method significantly (t = 117; p = 0) improved the overlap of the cortical regions (Dice increased by 57%). We concluded our method can significantly improve the registration accuracy of brain cortical regions.

Keywords: fMRI analysis, Brain, Functional imaging (e.g. fMRI)
Abstract: Functional connectivity (FC) has been widely investigated to understand the cognition and behavior that emerge from human brain. Recently, there is overwhelming evidence showing that quantifying temporal changes in FC may provide greater insight into fundamental properties of brain network. However, scant attentions has been given to characterize the functional dynamics of network organization. To address this challenge, we propose a novel spatio-temporal hub identification method for functional brain networks by simultaneously identifying hub nodes in each static sliding window and maintaining the reasonable dynamics across the sliding windows, which allows us to further characterize the full-spectrum evolution of hub nodes along with the subject-specific functional dynamics. We have evaluated our spatio-temporal hub identification method on resting-state functional resonance imaging (fMRI) data from an obsessive-compulsive disease (OCD) study, where our new functional hub detection method outperforms current methods (without considering functional dynamics) in terms of accuracy and consistency.

Index Terms— Dynamic functional network, brain network, graph spectrum, hub node

Keywords: Computational Imaging, Functional imaging (e.g. fMRI), Diffusion weighted imaging
Abstract: Fiber clustering is a prerequisite step towards tract-based analysis for human brain, and it is very important to explain brain structure and function relationship. Over the last decade, it has been an open and challenging question as to what a reasonable clustering of fibers is. Specifically, the purpose of fiber clustering is to cluster the whole brain’s white matter fibers extracted from tractography into similar and meaningful fiber bundles, thus how to definite the “similar and meaningful” metric decides the performance and possible application of a fiber clustering method. In the past, researchers typically divided the fibers into anatomical or structural similar bundles, but rarely divided them according to functional meanings. In this work, we proposed a novel fiber clustering method by adopting the functional and structural information and combined them into the input of a deep convolutional autoencoder with embedded clustering, which can better extract and use the features within the data. The experimental results show that the proposed method can cluster the whole brain’s fibers into functionally and structurally meaningful bundles.

Keywords: Fetus, Magnetic resonance imaging (MRI), Machine learning
Abstract: Understanding human fetal neurodevelopment is of great clinical importance as abnormal development is linked to adverse neuropsychiatric outcomes after birth. With the advances in functional Magnetic Resonance Imaging (fMRI), recent stud- ies focus on brain functional connectivity and have provided new insight into development of the human brain before birth. Deep Convolutional Neural Networks (CNN) have achieved remarkable success on learning directly from image data, yet have not been applied on fetal fMRI for understanding fetal neurodevelopment. Here, we bridge this gap by applying a novel application of 3D CNN to fetal blood oxygen-level dependence (BOLD) resting-state fMRI data. We build supervised CNN to isolate variation in fMRI signals that relate to younger v.s. older fetal age groups. Sensitivity analysis is then performed to identify brain regions in which changes in BOLD signal are strongly associated with fetal brain age. Based on the analysis, we discovered that regions that most strongly differentiate groups are largely bilateral, share similar distribution in older and younger age groups, and are areas of heightened metabolic activity in early human development.

Keywords: Magnetic resonance imaging (MRI), Brain, Machine learning
Abstract: We introduce a fast model based deep learning approach for calibrationless parallel MRI reconstruction. The proposed scheme is a non-linear generalization of structured low rank (SLR) methods that self learn linear annihilation filters from the same subject; the proposed scheme pre-learns the nonlinear annihilation relations in the Fourier domain from exemplar data. The pre-learning strategy significantly reduces the computational complexity, making the proposed scheme three orders of magnitude faster than SLR schemes. The proposed framework also allows the use of a complementary spatial domain prior; the hybrid regularization scheme offers improved performance over calibrated image domain MoDL approach.The calibrationless strategy minimizes potential mismatches between calibration data and main scan, while eliminating the need for a fully sampled calibration region.

Keywords: Magnetic resonance imaging (MRI), Brain
Abstract: This paper proposes an enhanced denoising autoencoder prior (EDAEP) learning framework for accurate multi-contrast MR image reconstruction. A multi-model structure with various noise levels is designed to capture features of different scales from different contrast images. Furthermore, a weighted aggregation strategy is proposed to balance the impact of different model outputs, making the performance of the proposed model more robust and stable while facing noise attacks. The model was trained to handle three different sampling patterns and different acceleration factors on two public datasets. Results demonstrate that our proposed method can improve the quality of reconstructed images and outperform the previous state-of-the-art approaches. The code is available at https://github.com/yqx7150.

Keywords: Diffusion weighted imaging, Compressive sensing & sampling, Magnetic resonance imaging (MRI)
Abstract: Conventional single-shell diffusion MRI experiments acquire sampled values of the diffusion signal from the surface of a sphere in q-space. However, to reduce data acquisition time, there has been recent interest in using regularization to enable q-space undersampling. Although different regularization strategies have been proposed for this purpose (i.e., sparsity-promoting of the spherical ridgelet representation and Laplace-Beltrami Tikhonov regularization), there has not been a systematic evaluation of the strengths, weaknesses, and potential synergies of the different regularizers. In this work, we use real diffusion MRI data to systematically evaluate the performance characteristics of these different approaches and determine whether one approach is fundamentally more powerful than the other. Results from retrospective subsampling experiments suggest that both regularization strategies offer largely similar reconstruction performance (though with different levels of computational complexity) with some degree of synergy (albeit, relatively minor).

Keywords: Magnetic resonance imaging (MRI), Image reconstruction - analytical & iterative methods, Machine learning
Abstract: The MRI reconstruction field lacked a proper data set that allowed for reproducible results on real raw data (i.e. complex-valued), especially when it comes to deep learning methods as this kind of methods require much more data than classical Compressed Sensing reconstruction. This lack is now filled by the fastMRI data set, and it is needed to evaluate recent deep learning models on this benchmark. Besides, these networks are written in different frameworks, in different repositories (if publicly available), it is therefore needed to have a common tool, publicly available, allowing a reproducible benchmark of the different methods and ease of building new models. We provide such a tool that allows the benchmark of different reconstruction deep learning models.

Keywords: Machine learning, Magnetic resonance imaging (MRI), Other-method
Abstract: HDR-MRI is a sophisticated non-linear image fusion technique for MRI which enhances image quality by fusing multiple contrast types into a single composite image. It offers improved outcomes in automatic segmentation and potentially in diagnostic power, but the existing technique is slow and requires accurate image co-registration in order to function reliably. In this work, a lightweight fully convolutional neural network architecture is developed with the goal of approximating HDR-MRI images in real-time. The resulting Contrast Enhancement Network (CEN) is capable of performing near-perfect (SSIM = 0.98) 2D approximations of HDR-MRI in 10ms and full 3D approximations in 1s, running two orders of magnitude faster than the original implementation. It is also able to perform the approximation (SSIM = 0.93) with only two of the three contrasts required to generate the original HDR-MRI image, while requiring no image co-registration.

Keywords: Magnetic resonance imaging (MRI), Modeling - Anatomical, physiological and pathological, Brain
Abstract: We propose a patient-specific arterial input function (AIF) and tracer kinetic (TK) model-driven network to rapidly estimate the extended Tofts-Kety kinetic model parameters in DCE-MRI. We term our network as AIF-TK-net, which maps an input comprising of an image patch of the DCE-time series and the patient-specific AIF to the output image patch of the TK parameters. We leverage the open-source NEURO-RIDER database of brain tumor DCE-MRI scans to train our network. Once trained, our model rapidly infers the TK maps of unseen DCE-MRI images on the order of a 0.34 sec/slice for a 256x256x65 time series data on a NVIDIA GeForce GTX 1080 Ti GPU. We show its utility on high time resolution DCE-MRI datasets where significant variability in AIFs across patients exists. We demonstrate that the proposed AIF-TK net considerably improves the TK parameter estimation accuracy in comparison to a network, which does not utilize the patient AIF.

Keywords: Magnetic resonance imaging (MRI), Magnetic resonance spectroscopy, Muscle
Abstract: Custom double-tuned birdcage coils were constructed to enable concurrent evaluation of a number of NMR indices in the golden retriever muscular dystrophy (GRMD) model of Duchenne muscular dystrophy (DMD). Seven rectus femoris muscle samples from dogs with ages ranging from 3 to 30 months were studied. 1H T1-weighted (T1w) and T2-weighted (T2w) images, 23Na images, and 31P spectra were acquired for each sample. 1H T1w and T2w images showed a decrease in T2w/T1w signal ratio for the four older (≥12 months) samples when compared to younger samples. Other NMR indices unexpectedly showed no significant correlation with age. The collection time of samples and varying levels of disease severity may have attributed to these results. Regardless, the associated custom coils and positioner developed to enable multi-nuclear studies will enable future work to investigate NMR-based biomarkers in the numerous GRMD samples available to our group.

Keywords: Computer-aided detection and diagnosis (CAD), Magnetic resonance spectroscopy, Brain
Abstract: Alzheimer’s disease (AD) is the most common form of dementia. Neuroimaging data is an integral part of the clinical assessment providing a way for clinicians to detect brain abnormalities for AD diagnosis. Anatomical MRI has been widely used to assess structural brain atrophy for AD detectionand prediction. In addition to structural changes, metabolic changes in some brain regions such as the Posterior Cingulate Cortex (PCC) could be a good bio-marker for an early AD detection. Recently, proton Magnetic Resonance Spectroscopy (1H-MRS) have been proved to be effective to reveal a wealth of brain metabolic information. In this paper, we propose an end-to-end deep leaning Network for early AD and Normal Control (NC) subjects classification using 1H-MRS raw data from the PCC area. This work is the first investigation of 1H-MRS data with deep-learning technique for early AD detection. Data of 135 subjects, collected in Poitiers university hospital, are used to learn the proposed DeepMRS network. Our classification of patients with early AD versus NC subjects achieves an AUC of 94,74%, a sensitivity of 100% and a specificity of 89,47% demonstrating a promising early dementia detection performance.

Keywords: Computer-aided detection and diagnosis (CAD), Gastrointestinal tract, Magnetic resonance imaging (MRI)
Abstract: It is vital to automatically detect the Extramural Vascular Invasion (EMVI) in rectal cancer before surgery, which facilitates to guide the patient's treatment planning. Nevertheless, there are few studies about EMVI detection through magnetic resonance imaging (MRI). Moreover, since EMVI has three main characteristics: highly-variable appearances, relatively-small sizes and similar shapes with surrounding tissues, current deep learning based methods can not be directly used. In this paper, we propose a novel and efficient EMVI detection framework, which gives rise to three main contributions. Firstly, we introduce a self-attention module to capture dependencies ranging from local to global. Secondly, we design a parallel atrous convolution (PAC) block and a global pyramid pooling (GPP) module to encode richer context information at multiple scales. Thirdly, we fuse the whole-scene and local-region information together to improve the feature representation ability. Experimental results show that our framework can significantly improve the detection accuracy and outperform other state-of-the-art methods.

Keywords: Ultrasound, Breast, Image segmentation
Abstract: Breast tumor segmentation provides accurate tumor boundary, and serves as a key step toward further cancer quantification. Although deep learning-based approaches have been proposed and achieved promising results, existing approaches have difficulty in detecting small breast tumors. The capacity to detecting small tumors is particular-ly important in finding early stage cancers using computer-aided diagnosis (CAD) systems. In this paper, we propose a novel deep learning architecture called Small Tumor-Aware Network (STAN), to improve the performance of segmenting tumors with different size. The new architecture integrates both rich context information and high-resolution image features. We validate the proposed approach using seven quantitative metrics on two public breast ultrasound datasets. The proposed approach outperformed the state-of-the-art approaches in segmenting small breast tumors.

Keywords: Computed tomography (CT), Machine learning
Abstract: Acute aortic syndrome (AAS) is a group of life threatening conditions of the aorta. We have developed an end-to-end automatic approach to detect AAS in computed tomography (CT) images. Our approach consists of two steps. At first, we extract N cross sections along the segmented aorta centerline for each CT scan. These cross sections are stacked together to form a new volume which is then classified using two different classifiers, a 3D convolutional neural network (3D CNN) and a multiple instance learning (MIL). We trained, validated, and compared two models on 2291 contrast CT volumes. We tested on a set aside cohort of 230 normal and 50 positive CT volumes. Our models detected AAS with an Area under Receiver Operating Characteristic curve (AUC) of 0.965 and 0.985 using 3DCNN and MIL, respectively.

Keywords: Computer-aided detection and diagnosis (CAD), Machine learning, Abdomen
Abstract: The diagnosis of the presence of metastatic lymph nodes from abdominal computed tomography (CT) scans is an essential task performed by radiologists to guide radiation and chemotherapy treatment. State-of-the-art deep learning classifiers trained for this task usually rely on a training set containing CT volumes and their respective image-level (i.e., global) annotation. However, the lack of annotations for the localisation of the regions of interest (ROIs) containing lymph nodes can limit classification accuracy due to the small size of the relevant ROIs in this problem. The use of lymph node ROIs together with global annotations in a multi-task training process has the potential to improve classification accuracy, but the high cost involved in obtaining the ROI annotation for the same samples that have global annotations is a roadblock for this alternative. We address this limitation by introducing a new training strategy from two data sets: one containing the global annotations, and another (publicly available) containing only the lymph node ROI localisation. We term our new strategy semi-supervised multi-domain multi-task training, where the goal is to improve the diagnosis accuracy on the globally annotated data set by incorporating the ROI annotations from a different domain. Using a private data set containing global annotations and a public data set containing lymph node ROI localisation, we show that our proposed training mechanism improves the area under the ROC curve for the classification task compared to several training method baselines.

Keywords: Computer-aided detection and diagnosis (CAD), Magnetic resonance imaging (MRI), Prostate
Abstract: The purpose of this study was to develop a quantitative method for detection and segmentation of clinically significant (ISUP grade ≥ 2) prostate cancer (PCa) in low-risk patient. A consecutive cohort of 356 patients (active surveillance) was selected and divided in two groups: 1) MRI and targeted-biopsy positive PCa, 2) MRI and standard-biopsy negative PCa. A 3D convolutional neural network was trained in three-fold cross validation with MRI and targeted-biopsy positive patient’s data using two mp-MRI sequences (T2-weighted, DWI-b800) and ADC map as input. After training, the model was tested on separate positive and negative patients to evaluate the performance. The model achieved an average area under the curve (AUC) of the receiver operating characteristics is 0.78 (sensitivity = 85%, specificity = 72%). The diagnostic performance of the proposed method in segmenting significant PCa and to conform non-significant PCa in low-risk patients is characterized by a good AUC and negative-predictive-value.

Keywords: Computer-aided detection and diagnosis (CAD), Breast, X-ray imaging
Abstract: Retrieving cases with similar image features has been found to be effective for improving the diagnostic accuracy of microcalcification (MC) lesions in mammograms. However, a major challenge in such an image-retrieval approach is how to determine a retrieved lesion image has diagnostically similar features to that of a query case. We investigate the feasibility of modeling perceptually similar MC lesions by using deep learning features extracted from two types of deep neural networks, of which one is a supervised-learning network developed for the task of MC detection and the other is a denoising autoencoder network. In the experiments, the deep learning features were compared against the perceptual similarity scores collected from a reader study on 1,000 MC lesion image pairs. The results indicate that the deep learning features can potentially be more effective for modelling the notion of perceptual similarity of MC lesions than traditional handcrafted texture features.

Keywords: Machine learning, Modeling - Knowledge, Other-method
Abstract: Access to hospital data is commonly a difficult, costly and time-consuming process requiring extensive interaction with network administrators. This leads to possible delays in obtaining insights from data, such as diagnosis or other clinical outcomes. Healthcare administrators, medical practitioners, researchers and patients could benefit from a system that could extract relevant information from healthcare data in real-time. In this paper, we present a question answering system that allows health professionals to interact with a large-scale database by asking questions in natural language. This system is built upon the BERT and SQLOVA models, which translate a user's request into an SQL query, which is then passed to the data server to retrieve relevant information. We also propose a deep bilinear similarity model to improve the generated SQL queries by better matching terms in the user's query with the database schema and contents. This system was trained with only 75 real questions and 455 back-translated questions, and was evaluated over 75 additional real questions about a real health information database, achieving a retrieval accuracy of 78%.

Keywords: Magnetic resonance imaging (MRI), Machine learning, Motion compensation and analysis
Abstract: Motion is one of the main sources for artifacts in magnetic resonance (MR) images. It can have significant consequences on the diagnostic quality of the resultant scans. Previously, supervised adversarial approaches have been suggested for the correction of MR motion artifacts. However, these approaches suffer from the limitation of required paired co-registered datasets for training which are often hard or impossible to acquire. Building upon our previous work, we introduce a new adversarial framework with a new generator architecture and loss function for the unsupervised correction of severe rigid motion artifacts in the brain region. Quantitative and qualitative comparisons with other supervised and unsupervised translation approaches showcase the enhanced performance of the introduced framework.

Keywords: Image segmentation, Brain, Magnetic resonance imaging (MRI)
Abstract: Brain lesion segmentation is crucial for diagnosis, surgical planning, and analysis. Owing to the fact that pixel values of brain lesions in magnetic resonance (MR) scans are distributed over the wide intensity range, there is always a considerable overlap between the class-conditional densities of lesions. Hence, an accurate automatic brain lesion segmentation is still a challenging task. We present a novel architecture based on conditional generative adversarial networks (cGANs) to improve the lesion contrast for segmentation. To this end, we propose a novel generator adaptively calibrating the input pixel values, and a Markovian discriminator to estimate the distribution of tumors. We further propose the Enhancement and Segmentation GAN (Enh-Seg-GAN) which effectively incorporates the classifier loss into the adversarial one during training to predict the central labels of the sliding input patches. Particularly, the generated synthetic MR images are a substitute for the real ones to maximize lesion contrast while suppressing the background. The potential of proposed frameworks is confirmed by quantitative evaluation compared to the state-of-the-art methods on BraTS'13 dataset.

Keywords: Image segmentation, Image synthesis, Vessels
Abstract: Traditional segmentation methods are annotation-free but usually produce unsatisfactory results. The latest leading deep learning methods improve the results but require expensive and time-consuming pixel-level manual annotations. In this work, we propose a novel method based on self-supervision and generative adversarial network (GAN), which has high performance and requires no manual annotations. First, we perform traditional segmentation methods to obtain coarse segmentation. Then, we use GAN to generate a synthetic image, on which the image foreground is pixel-to-pixel corresponding to the coarse segmentation. Finally, we train the segmentation model with the data pairs of synthetic images and coarse segmentations. We evaluate our method on two types of segmentation tasks, including red blood cell (RBC) segmentation on microscope images and vessel segmentation on digital subtraction angiographies (DSA). The results show that our annotation-free method provides a considerable improvement over the traditional methods and achieves comparable accuracies with fully supervised methods.

Keywords: Image segmentation
Abstract: Convolutional neural networks (CNNs) have been successfully applied to medical image classification, segmentation, and related tasks. Among the many CNNs architectures, U-Net and its improved versions based are widely used and achieve state-of-the-art performance these years. These improved architectures focus on structural improvements and the size of the convolution kernel is generally fixed. In this paper, we propose a module that combines the benefits of multiple kernel sizes and apply it to U-Net its variants. We test our module on three segmentation benchmark datasets and experimental results show significant improvement.

Keywords: Image segmentation, Ultrasound, Eye
Abstract: This work explores a hybrid approach to segmentation as an alternative to a purely data driven approach. We introduce an end-to-end U-Net based network called DU-Net, which uses additional frequency preserving features, namely the Scattering Coefficients (SC) for medical image segmentation. SC are translation invariant and Lipschitz continuous to deformations which help DU-Net outperform other conventional CNN counterparts on four datasets and two segmentation tasks: Optic Disc and Optic Cup in color fundus images and fetal Head in ultrasound images. The proposed method shows remarkable improvement over the basic U-Net with performance competitive to state-of-the-art methods. The results indicate that it is possible to use a lighter network trained with fewer images (without any augmentation) to attain good segmentation results.

Keywords: Computer-aided detection and diagnosis (CAD), Classification, Computed tomography (CT)
Abstract: Deep learning systems are now being widely used to analyze lung cancer. However, recent work has shown a deep learning system can be easily fooled by intentionally adding some noise in the image. This is called as Adversarial attack. This paper presents an adversarial attack for malignancy prediction of lung nodules. We found that the adversarial attack can cause significant changes in lung nodule malignancy prediction accuracy. An ensemble-based defense strategy was developed to reduce the effect of an adversarial attack. A multi-initialization based CNN ensemble was utilized. We also explored adding adversarial images in the training set, which eventually reduced the rate of mis-classification and made the CNN models more robust to an adversarial attack. A subset of cases from the National Lung Screening Trial (NLST) dataset were used in our study. Initially, 75.1%, 75.5% and 76% classification accuracy were obtained from the three CNNs on original images (without an adversarial attack). Fast Gradient Sign Method (FGSM) and one-pixel attacks were analyzed. After the FGSM attack, 46.4%, 39.24%, and 39.71% accuracy was obtained from the 3 CNNs. Whereas, after a one pixel attack 72.15%, 73%, and 73% classification accuracy was achieved. FGSM caused much more damaged to CNN prediction. With a multi-initialization based ensemble and including adversarial images in the training set, 82.27% and 81.43% classification accuracy were attained after FGSM and one-pixel attacks respectively.

Keywords: Image segmentation, Machine learning, Computed tomography (CT)
Abstract: Accurate segmentation of organ at risk (OARs) in head and neck CT images is critical for planning of radiotherapy of the nasopharynx cancer. In segmentation tasks, fully convolutional networks (FCNs) are widely used. Recently, as a kind of attention module, concurrent squeeze and excitation (scSE) blocks in FCNs are proved to have good performance. However, the attention feature maps generated by scSE blocks are isolated from each other, which doesn’t help network notice the similarities among different feature maps. Consequently, we propose cross-layer spatial attention map fusion network (CSAF-CNN) to fuse different spatial attention maps to solve this problem. In addition, we introduce a top-k exponential logarithmic dice loss (TELD-Loss) in OARs segmentation, which effectively alleviates the serious sample imbalance problem of this task. We evaluate our framework in the head & neck CT scans of nasopharynx cancer patients in StructSeg 2019 challenge. We validate the effectiveness of the proposed method through ablation study, and achieve very competitive results.

Keywords: X-ray imaging, Bone, Machine learning
Abstract: Acute Proximal Femoral Fractures are a growing health concern among the aging population. These fractures are often associated with significant morbidity and mortality as well as reduced quality of life. Furthermore, with the increasing life expectancy owing to advances in healthcare, the number of proximal femoral fractures may increase by a factor of 2 to 3, since the majority of fractures occur in patients over the age of 65. In this paper, we show that by using transfer learning and leveraging pre-trained models, we can achieve very high accuracy in detecting fractures and that they can be localized utilizing class activation maps.

Keywords: Diffusion weighted imaging, Brain, Machine learning
Abstract: Learning-based approaches, especially those based on deep networks, have enabled high-quality estimation of tissue microstructure from low-quality diffusion magnetic resonance imaging (dMRI) scans, which are acquired with a limited number of diffusion gradients and a relatively poor spatial resolution. These learning-based approaches to tissue microstructure estimation require acquisitions of training dMRI scans with high-quality diffusion signals, which are densely sampled in the q-space and have a high spatial resolution. However, the acquisition of training scans may not be available for all datasets. Therefore, we explore knowledge transfer between different dMRI datasets so that learning-based tissue microstructure estimation can be applied for datasets where training scans are not acquired. Specifically, for a target dataset of interest, where only low-quality diffusion signals are acquired without training scans, we exploit the information in a source dMRI dataset acquired with high-quality diffusion signals. We interpolate the diffusion signals in the source dataset in the q-space using a dictionary-based signal representation, so that the interpolated signals match the acquisition scheme of the target dataset. Then, the interpolated signals are used together with the high-quality tissue microstructure computed from the source dataset to train deep networks that perform tissue microstructure estimation for the target dataset. Experiments were performed on brain dMRI scans with low-quality diffusion signals, where the benefit of the proposed strategy is demonstrated.

Keywords: Pattern recognition and classification, Magnetic resonance imaging (MRI), Brain
Abstract: Current physio-pathological data suggest that Parkinson's Disease (PD) symptoms are related to important alterations in subcortical brain structures. However, structural changes in these small structures remain difficult to detect for neuro-radiologists, in particular, at the early stages of the disease ('de novo' PD patients). The absence of a reliable ground truth at the voxel level prevents the application of traditional supervised deep learning techniques. In this work, we consider instead an anomaly detection approach and show that auto-encoders (AE) could provide an efficient anomaly scoring to discriminate 'de novo' PD patients using quantitative Magnetic Resonance Imaging (MRI) data.

Keywords: Image segmentation, Liver
Abstract: In this work we address the problem of domain adaptation for segmentation tasks with deep convolutional neural networks. We focus on managing the domain shift from MRI to CT volumes on the example of 3D liver segmentation. Domain adaptation between modalities is particularly of practical importance, as different hospital departments usually tend to use different imaging modalities and protocols in their clinical routine. Thus, training a model with source data from one department may not be sufficient for application in another institution. Most adaptation strategies make use of target domain samples and often additionally incorporate the corresponding ground truths from the target domain during the training process. In contrast to these approaches, we investigate the possibility of training our model solely on source domain data sets, i.e. we apply zero-shot domain adaptation. To compensate the missing target domain data, we use prior knowledge about both modalities to steer the model towards more general features during the training process. We particularly make use of fixed Sobel kernels to enhance contour information and apply anatomical priors, learned separately by a convolutional autoencoder. Although we completely discard including the target domain in the training process, our proposed approach improves a vanilla U-Net implementation drastically and yields promising segmentation results.

Keywords: Machine learning, Pattern recognition and classification, Computed tomography (CT)
Abstract: Deep neural networks represent the state of the art for computer-aided medical imaging assessment, e.g. lesion detection, organ segmentation and disease classification. While for large datasets their superior performance is a clear argument, medical imaging data is often small and highly heterogeneous. In combination with the typical parameter amount in deep neural networks, this often leads to overfitting and results in a low level of generalization performance. We propose a straight-forward combination of random forests and deep neural networks for superior performance on medical imaging datasets with only small data, and provide an extensive evaluation of survival prediction for metastatic colorectal cancer patients using computed tomography imaging data, with our proposed method clearly outperforming other approaches.

Keywords: Dimensionality reduction, Machine learning, fMRI analysis
Abstract: Conventional methods designed for Mild Cognitive Impairment (MCI) classification usually assume that the MCI subjects are homogeneous. However, recent discoveries indicate that MCI has heterogeneous neuropathological origins which may contribute to the sub-optimal performance of conventional methods. To compensate for the limitations of existing methods, we propose Maximum Margin Heterogeneous Feature Selection (MMHFS) by explicitly considering the heterogeneous distribution of MCI data. More specifically, the proposed method simultaneously performs unsupervised clustering discovery on MCI data and conducts discriminant feature selection to help classify MCI from Normal Control (NC). It is worth noting that these two processes can benefit from each other, thus enabling the proposed method to achieve better performance.

Keywords: Image segmentation, Liver, Computed tomography (CT)
Abstract: Liver vessel segmentation from contrast CT images is critical in liver surgical planning and navigation. Due to the complex vessel systems, manual segmentation of liver vessel is laborious and error-prone. In this work we propose liver vessel segmentation deep neural networks which are able to perform the segmentation automatically. Dense image labelling incurred by normal deep learning based segmentation makes it impractical for vessel segmentation. On the contrary, our method only requires a few of initial voxel-level labels, and thus tremendously reduces the workload of manual annotating. A small number of liver vessels are firstly annotated and used to train a sparse dictionary and logistic regressor to get the preliminary vessel predictions. Then the proposed deep neural network is trained based on bootstrapping technique where the training target is a convex combination of the model predictions and those preliminary vessel predictions. Followed by the post-processing step consisting of region connection and noisy removal, the liver vessel tree is finally built from model predictions. Experimental results of3DIRCADb dataset show that the proposed method can effectively extract liver vessel trees from contrast CT images with minimum annotations.The state-of-art results have been achieved on this small dataset with the average dice and sensitivity 0.687±0.041 and 0.786±0.105 respectively.

Keywords: Computer-aided detection and diagnosis (CAD), Liver, Magnetic resonance imaging (MRI)
Abstract: The slice thickness of MR imaging may remarkably degrade the clarity of 3D lesion images within through-plane slices (coronal or sagittal views) so as to influence the performance of lesion characterization. To alleviate the problem, we propose an end-to-end super-resolution and self-attention framework based on Generative adversarial networks (GAN) for improving the malignancy characterization of hepatocellular carcinoma (HCC). Specifically, a super-resolution subnetwork is designed to enhance the low-resolution patches of coronal or sagittal views based on texture transferred from the high-resolution patches of the axial view, and then the enhanced patches are fed into the classification subnetwork for malignancy characterization. Furthermore, a self-attention mechanism is utilized to extract multiple discriminative features for better super-resolution and lesion characterization. Experimental results of clinical HCCs demonstrate the superior performance of the proposed method compared with conventional CNN-based methods and show the potential in clinical practice.

Keywords: Optical coherence tomography, Image quality assessment, Eye
Abstract: Optical coherence tomography (OCT) has become an important tool for the diagnosis of retinal diseases, and image quality assessment on OCT images has considerable clinical significance for guaranteeing the accuracy of diagnosis by ophthalmologists. Traditional OCT image quality assessment is usually based on hand-crafted features including signal strength index and signal to noise ratio. These features only reflect a part of image quality, but cannot be seen as a full representation on image quality. Especially, there is no detailed description of OCT image quality so far. In this paper, we firstly define OCT image quality as three grades (‘Good’, ‘Usable’ and ‘Poor’). Considering the diversity of image quality, we then propose a deep and shallow features fusion network (DSFF-Net) to conduct multiple label classification. The DSFF-Net combines deep and enhanced shallow features of OCT images to predict the image quality grade. The experimental results on a large OCT dataset show that our network obtains state-of-the-art performance, outperforming the other classical CNN networks.

Keywords: Image enhancement/restoration(noise and artifact reduction), Optical coherence tomography, Eye
Abstract: Optical coherence tomography (OCT) is a micrometer-resolution, cross-sectional imaging modality for biological tissue. It has been widely applied for retinal imaging in ophthalmology. However, the large speckle noise affects the analysis of OCT retinal images and their diagnostic utility. In this article, we present a new speckle reduction algorithm for 3D OCT images. The OCT speckle noise is approximated as Poisson distribution, which is difficult to be removed for its signal-dependent characteristic. Thus our algorithm is consisted by two steps: first, a variance-stabilizing trans-formation, named Anscombe transformation, is applied to redistribute the multiplicative speckle noise into an additive Gaussian noise; then the transformed data is decomposed and filtered in 3D Shearlet domain, which provides better representation of the edge information of the retinal layers than wavelet and curvelet. The proposed method is evaluated through the three parameters using high-definition B-scans as the ground truth. Quantitative experimental results show that our method gives out the best evaluation parameters, and highest edge contrast, compared with state-of-the-art OCT denoising algorithms.

Keywords: Optical coherence tomography, Computer-aided detection and diagnosis (CAD), Eye
Abstract: Optical coherence tomography (OCT) is widely used in computer-aided medical diagnosis of retinal pathologies. Deep convolutional network has been successfully applied to detect lesions from OCT images. Different OCT imaging devices inevitably cause variation in the distribution between training phase and testing phase, which will lead to extremely reduction on model performance. Most existing unsupervised domain adaptation methods are mainly focused on lesion segmentation, there are few studies on lesion detection tasks especially for OCT images. In this paper, we propose a novel unsupervised domain adaptation framework adaptively learning feature representation to achieve cross-device lesion detection for OCT images. Firstly, we design global and local adversarial discriminators to force the networks to learn device-independent features. Secondly, we develop a non-parameter adaptive feature norm into global adversarial discriminator to stabilize the discrimination in target domain. Finally, we perform the validation experiment on lesion detection task across two OCT devices. The results exhibit that the proposed framework has promising performance compared with other unsupervised domain adaptation approaches.

Keywords: Machine learning, Eye, Optical coherence tomography
Abstract: Macular ganglion cell inner plexiform layer (GCIPL) thickness is an important biomarker for clinical managements of glaucoma. Clinical analysis of GCIPL progression uses averaged thickness only, which easily washes out small changes and reveals no spatial patterns. This is the first work to predict the 2D GCIPL thickness map. We propose a novel Time-aware Convolutional Long Short-Term Memory (TC-LSTM) unit to decompose memories into the short-term and long-term memories and exploit time intervals to penalize the short-term memory. TC-LSTM unit is incorporated into an auto-encoder-decoder so that the end-to-end model can handle irregular sampling intervals of longitudinal GCIPL thickness map sequences and capture both spatial and temporal correlations. Experiments show the superiority of the proposed model over the traditional method.

Keywords: Optical coherence tomography, Eye, Image segmentation
Abstract: Anterior segment optical coherence tomography (AS-OCT) is widely used to observe the corneal tissue structures in clinical ophthalmology. Accurate segmentation of corneal tissue interfaces is essential for corneal diseases diagnosis and surgical planning. However, image scattered noise and keratopathy make corneal tissue interface fitting results of the existing methods deviate. In this paper, we propose a hierarchy-constrained network, which combines hierarchical features by using of an elegant progressive feature-extraction module and boundary constraint to overcome these challenges. In the meantime, multi-level prediction fusion module is integrated to the network that eventually enables the output node to sufficiently absorb features extracted from various levels. Extensive experimental results on two datasets containing cornea images with multiple lesions show that our proposed method distinctly improves the accuracy of corneal tissue interfaces segmentation and outperforms other existing methods.

Keywords: Optical coherence tomography, Machine learning, Elasticity measures
Abstract: Mechanical properties of tissue provide valuable information for identifying lesions. One approach to obtain quantitative estimates of elastic properties is shear wave elastography with optical coherence elastography (OCE). However, given the shear wave velocity, it is still difficult to estimate elastic properties. Hence, we propose deep learning to directly predict elastic tissue properties from OCE data. We acquire 2D images with a frame rate of 30 kHz and use convolutional neural networks to predict gelatin concentration, which we use as a surrogate for tissue elasticity. We compare our deep learning approach to predictions from conventional regression models, using the shear wave velocity as a feature. Mean absolut prediction errors for the conventional approaches range from 1.32+-0.98 p.p. to 1.57+-1.30 p.p. whereas we report an error of 0.90+-0.84 p.p. for the convolutional neural network with 3D spatio-temporal input. Our results indicate that deep learning on spatio-temporal data outperforms elastography based on explicit shear wave velocity estimation.

Keywords: Microscopy - Light, Confocal, Fluorescence, In-vivo cellular and molecular imaging, Image segmentation
Abstract: The reservoir computing (RC) paradigm is exploited to detect Jurkat T cells and antibody-coated beads in fluorescence microscopy data. Recent progress in imaging of subcellular calcium (Ca²⁺) signaling offers a high spatial and temporal resolution to characterize early signaling events in T cells. However, data acquisition with dual-wavelength Ca²⁺ indicators, the photo-bleaching at high acquisition rate, low signal-to-noise ratio, and temporal fluctuations of image intensity entail corporation of post-processing techniques into Ca²⁺ imaging systems. Besides, although continuous recording enables real-time Ca²⁺ signal tracking in T cells, reliable automated algorithms must be developed to characterize the cells, and to extract the relevant information for conducting further statistical analyses. Here, we present a robust two-channel segmentation algorithm to detect Jurkat T lymphocytes as well as antibody-coated beads that are utilized to mimic cell-cell interaction and to activate the T cells in microscopy data. Our algorithm uses the reservoir computing framework to learn and recognize the cells -- taking the spatiotemporal correlations between pixels into account. A comparison of segmentation accuracy on testing data between our proposed method and the deep learning U-Net model confirms that the developed model provides accurate and computationally cheap solution to the cell segmentation problem.

Keywords: Molecular and cellular screening, Microscopy - Light, Confocal, Fluorescence, Image compression
Abstract: Bacteria entering the bloodstream causes bloodstream infection (BSI). Without proper treatment, BSI can lead to sepsis which is a life-threatening condition. Detection of bacteria in blood at the early stages of BSI can effectively prevent the development of sepsis. Using microfluidic droplets for single bacterium encapsulation provides single-digit bacterial detection sensitivity. In this study, samples of ampicillin-resistant E. coli in human blood were partitioned into millions of 30 μm diameter microfluidic droplets and followed by 8-hour culturing. Thousands of fluorescent bacteria from a single colony filled up the positive droplets after the culturing process. A circle detection software based on Hough Transform was developed to count the number of positive droplets from fluorescence images. The period to process one image can be as short as 0.5 ms when the original image is pre-processed and binarized by the developed software.

Keywords: Image enhancement/restoration(noise and artifact reduction), Microscopy - Super-resolution, Cells & molecules
Abstract: Single-molecule localization microscopy (SMLM) is a super-resolution imaging technique developed to image structures smaller than the diffraction limit. This modality results in sparse and non-uniform sets of localized blinks that need to be reconstructed to obtain a super-resolution representation of a tissue. In this paper, we explore the use of the Noise2Noise (N2N) paradigm to reconstruct the SMLM images. Noise2Noise is an image denoising technique where a neural network is trained with only pairs of noisy realizations of the data instead of using pairs of noisy/clean images, as performed with Noise2Clean (N2C). Here we have adapted Noise2Noise to the 2D SMLM reconstruction problem, exploring different pair creation strategies (fixed and dynamic). The approach was applied to synthetic data and to real 2D SMLM data of actin filaments. This revealed that N2N can achieve reconstruction performances close to the Noise2Clean training strategy, without having access to the super-resolution images. This could open the way to further improvement in SMLM acquisition speed and reconstruction performance.

Keywords: Image synthesis, Cells & molecules, Image enhancement/restoration(noise and artifact reduction)
Abstract: Data augmentation is a popular technique with which new dataset samples are artificially synthesized to the end of aiding training of learning-based algorithms and avoiding overfitting. Methods based on Generative adversarial networks (GANs) have recently rekindled interest in research on new techinques for data augmentation. With the current paper we propose a new GAN-based model for data augmentation, comprising a suitable Markov Random Field-based spatial constraint that encourages synthesis of spatially smooth outputs. Oriented towards use with medical imaging sets where a localization/segmentation annotation is available, our model can simultaneously also produce artificial annotations. We gauge performance numerically by measuring performance of U-Net trained to detect cells on microscopy images, by taking into account the produced augmented dataset. Numerical trials, as well as qualitative results validate the usefulness of our model.

Keywords: Machine learning, Classification, Microscopy - Light, Confocal, Fluorescence
Abstract: We present ESCELL, a method for developing an emergent symbolic language of communication between multiple agents reasoning about cells. We show how agents are able to cooperate and communicate successfully in the form of symbols similar to human language to accomplish a task in the form of a referential game (Lewis’ signaling game). In one form of the game, a sender and a receiver observe a set of cells from 5 different cell phenotypes. The sender is told one cell is a target and is allowed to send one symbol to the receiver from a fixed arbitrary vocabulary size. The receiver relies on the information in the symbol to identify the target cell. We train the sender and receiver networks to develop an innate emergent language between themselves to accomplish this task. We observe that the networks are able to successfully identify cells from 5 different phenotypes with an accuracy of 93.2%. We also introduce a new form of the signaling game where the sender is shown one image instead of all the images that the receiver sees. The networks successfully develop an emergent language to get an identification accuracy of 77.8%.

Keywords: Microscopy - Light, Confocal, Fluorescence, Cells & molecules, Image synthesis
Abstract: The image-based simulations in biomedicine play an important role as the real image data are difficult to be fully and precisely annotated. An increasing capability of contemporary computers allows to model and simulate reasonably complicated structures and in the last years also the dynamic processes. In this paper, we introduce a complex 3D model that describes the structure and dynamics of the population of endothelial cells. The model is based on standard cellular Potts model. It describes the formation process of a complex tubular network of endothelial cells fully in 3D together with the simulation of the cell death called apoptosis. The generated network imitates the structure and behavior that can be observed in real phase-contrast microscopy. The generated image data may serve as a benchmark dataset for newly designed detection or tracking algorithms.

Keywords: Microscopy - Light, Confocal, Fluorescence, Visualization, Quantification and estimation
Abstract: Recent advancements in biomedical imaging focus on fluorescent imaging using laser scanning confocal microscopy. However, high-resolution imaging of cellular activity remains considerably expensive for both in vitro and in vivo model. In this context, integration of mathematical modeling and imaging data analysis to predict the cellular activity may aid understanding of cell signaling. Here we performed dynamic imaging using confocal microscopy and propose a model considering cell to cell connectivity that can predict the effect of drug on Ca2+ oscillations. The proposed model consists of large number of ordinary differential (ODE) equations and uses the concept of adjoint matrix containing coupling factors to capture the activity of cells with random arrangement. The results show that the cell-to-cell connection plays a crucial role in controlling the calcium oscillations through a diffusion-based mechanism. The present simulation tool can be used as generalized framework to generate and visualize the time-lapse videos required for in vitro drug testing for various drug doses.

Keywords: Microscopy - Light, Confocal, Fluorescence, Single cell & molecule detection, Machine learning
Abstract: Visualization and classification of cell cycle stages in live cells requires the introduction of transient or stably expressing fluorescent markers. This is not feasible for all cell types, and can be time consuming to implement. Labelling of living cells also has the potential to perturb normal cellular function. Here we describe a computational strategy to estimate core cell cycle stages without markers by taking advantage of features extracted from information-rich ptychographic time-lapse movies. We show that a deep-learning approach can estimate the cell cycle trajectories of individual human melanoma cells from short 3-frame (~23 minute) snapshots, and can identify cell cycle arrest induced by chemotherapeutic agents targeting melanoma driver mutations.

Keywords: Microscopy - Light, Confocal, Fluorescence, Deconvolution, Image acquisition
Abstract: A predictive theoretical model based on Cramer-Rao bound for the uncertainty in fluorescent protein mixture composition detection has been developed. A physically based representation of the experimental efficiency through the Cauchy-Binet identity is presented. This approach allows the optimal design of the experiments without the introduction of artificial figures of merit.

Keywords: Microscopy - Light, Confocal, Fluorescence, Single cell & molecule detection, Pattern recognition and classification
Abstract: Cell migration is pivotal for their development, physiology and disease treatment. A single cell on a 2D surface can utilize continuous or discontinuous migration modes. To comprehend the cell migration, an adequate quantification for single cell-based analysis is crucial. An automatized approach could alleviate tedious manual analysis, facilitating large-scale drug screening. Supervised deep learning has shown promising outcomes in computerized microscopy image analysis. However, their implication is limited due to the scarcity of carefully annotated data and uncertain deterministic outputs. We compare three deep learning models to study the problem of learning discriminative morphological representations using weakly annotated data for predicting the cell migration modes. We also estimate Bayesian uncertainty to describe the confidence of the probabilistic predictions. Amongst three compared models, DenseNet yielded the best results with a sensitivity of 87.91% at a false negative rate of 1.26%.

Keywords: Classification, Tissue, Microscopy - Light, Confocal, Fluorescence
Abstract: Deep learning has proven to successfully learn variations in tissue and cell morphology. Training of such models typically relies on expensive manual annotations. Here we conjecture that spatially resolved gene expression, e.i., the transcriptome, can be used as an alternative to manual annotations. In particular, we trained five convolutional neural networks with patches of different size extracted from locations defined by spatially resolved gene expression. The network is trained to classify tissue morphology related to two different genes, general tissue, as well as background, on an image of fluorescence stained nuclei in a mouse brain coronal section. Performance is evaluated on an independent tissue section from a different mouse brain, reaching an average Dice score of 0.51. Results may indicate that novel techniques for spatially resolved transcriptomics together with deep learning may provide a unique and unbiased way to find genotype-phenotype relationships.

Keywords: Machine learning, Microscopy - Light, Confocal, Fluorescence, Quantification and estimation
Abstract: Accurate analysis of vesicle trafficking in live cells is challenging for a number of reasons: varying appearance, complex protein movement patterns, and imaging conditions. To allow fast image acquisition, we study how employing an astigmatism can be utilized for obtaining additional information that could make tracking more robust. We present two approaches for measuring the z position of individual vesicles.Firstly, Gaussian curve fitting with CNN-based denoising is applied to infer the absolute depth around the focal plane of each localized protein. We demonstrate that adding denoising yields more accurate estimation of depth while preserving the overall structure of the localized proteins. Secondly, we investigate if we can predict using a custom CNN architecture the axial trajectory trend. We demonstrate that this method performs well on calibration beads data without the need for denoising. By incorporating the obtained depth information into a trajectory analysis, we demonstrate the potential improvement in vesicle tracking.

Keywords: Endoscopy, Multi-modality fusion, Image registration
Abstract: Bronchoscopy and three-dimensional (3D) computed tomography (CT) are important complementary tools for managing lung diseases. Endobronchial video captured during bronchoscopy gives live views of the airway-tree interior with vivid detail of the airway mucosal surfaces, while the 3D CT images give considerable anatomical detail. Unfortunately, little effort has been made to link these rich data sources. This paper describes a rapid interactive multi-frame method for registering the video frames constituting a complete bronchoscopic video sequence onto their respective locations within the CT-based 3D airway tree. Registration results using both phantom and human cases show our method’s efficacy compared to ground-truth data, with a maximum position error = 8.5mm, orientation error of 17 degrees, and a minimum trajectory accuracy = 94.1%. We also apply our method to multimodal 3D airway-wall analysis within a comprehensive bronchoscopic video analysis system.

Keywords: Computer-aided detection and diagnosis (CAD), Endoscopy, Machine learning
Abstract: The automatic detection of frames containing polyps from a colonoscopy video sequence is an important first step for a fully automated colonoscopy analysis tool. Typically, such detection system is built using a large annotated data set of frames with and without polyps, which is expensive to be obtained. In this paper, we introduce a new system that detects frames containing polyps as anomalies from a distribution of frames from exams that do not contain any polyps. The system is trained using a one-class training set consisting of colonoscopy frames without polyps -- such training set is considerably less expensive to obtain, compared to the 2-class data set mentioned above. During inference, the system is only able to reconstruct frames without polyps, and when it tries to reconstruct a frame with polyp, it automatically removes (i.e., photoshop) it from the frame -- the difference between the input and reconstructed frames is used to detect frames with polyps. We name our proposed model as anomaly detection generative adversarial network (ADGAN), comprising a dual GAN with two generators and two discriminators. To test our framework, we use a new colonoscopy data set with 14317 images, split as a training set with 13350 images without polyps, and a testing set with 290 abnormal images containing polyps and 677 normal images without polyps. We show that our proposed approach achieves the state-of-the-art result on this data set, compared with recently proposed anomaly detection systems.

Keywords: Machine learning
Abstract: Dysphagia, or difficulty swallowing, is a serious health problem that reduces the quality of life of those affected. The standard method to diagnose dysphagia is the x-ray video fluoroscopic swallowing exam (VFSE). In this paper we investigate the use of deep learning networks to classify VFSE as normal or abnormal. The proposed network is based on a long term recurrent convolutional network (LRCN). This network was trained and validated using 1154 VFSE. Using 10-fold cross-validation, the accuracy of classification was 85% and the area under the ROC curve was 0.89. This work shows the promise of using deep learning networks as a screening tool to detect dysphagia in VFSE.

Keywords: Computational Imaging, Image reconstruction - analytical & iterative methods, Machine learning
Abstract: Single-pixel imaging allows low cost cameras to be built for imaging modalities where a conventional camera would either be too expensive or too cumbersome. This is very attractive for biomedical imaging applications based on hyperspectral measurements, such as image-guided surgery, which requires the full spectrum of fluorescence.

A single-pixel camera essentially measures the inner product of the scene and a set of patterns. An inverse problem has to be solved to recover the original image from the raw measurement. The challenge in single-pixel imaging is to reconstruct the video sequence in real time from under-sampled data.

Previous approaches have focused on the reconstruction of each frame independently, which fails to exploit the natural temporal redundancy in a video sequence. In this study, we propose a fast deep-learning reconstructor that exploits the spatio-temporal features in a video. In particular, we consider convolutional gated recurrent units that have low memory requirements. Our simulation shows than the proposed recurrent network improves the reconstruction quality compared to static approaches that reconstruct the video frames independently.

Keywords: Computer-aided detection and diagnosis (CAD), Machine learning, Gastrointestinal tract
Abstract: Automatic polyp detection is reported to have a high false-positive rate (FPR) because of various polyp-like structures and artifacts in complex colon environment. An efficient polyp’s computer-aided detection (CADe) system should have a high sensitivity and a low FPR (high specificity). Convolutional neural networks have been implemented in colonoscopy-based automatic polyp detection and achieved high performance in improving polyp detection rate. However, complex colon environments caused excessive false positives are going to prevent the clinical implementation of CADe system. To reduce false positive rate, we proposed an automatic polyp detection algorithm, combined with YOLOv3 architecture and active learning. This algorithm was trained with colonoscopy videos/ images from 283 subjects. Through testing with 100 short and 9 full colonoscopy videos, the proposed algorithm shown FPR of 2.8% and 1.5%, respectively, similar sensitivities of expert endoscopists.

Keywords: Microscopy - Light, Confocal, Fluorescence, Gastrointestinal tract, Machine learning
Abstract: Histologic diagnosis of Barrett's esophagus and esophageal malignancy via probe-based confocal laser endomicroscopy (pCLE) allows for real-time examination of epithelial architecture and targeted biopsy sampling. Although pCLE has demonstrated high specificity, sensitivity remains low. This study employs deep learning architectures in order to improve the accuracy of pCLE in diagnosing esophageal cancer and its precursors. pCLE videos are curated and annotated as belonging to one of the three classes: squamous, Barrett's (intestinal metaplasia without dysplasia), or dysplasia. We introduce two novel video architectures, AttentionPooling and Multi-Module AttentionPooling deep networks, that outperform other models and demonstrate a high degree of explainability.

Keywords: Magnetic resonance imaging (MRI), Brain, Image acquisition
Abstract: In MRI there has been no 3D+time radial view-order which meets all the desired characteristics for simultaneous dynamic/high-resolution imaging, such as for self-navigated motion-corrected high resolution neuroimaging.

In this work, we examine the use of Hierarchical Equal Area iso-Latitude Pixelization (HEALPix) for generation of three dimensional dynamic (3D+time) radial view-orders for MRI, and compare to a selection of commonly used 3D view-orders.

The resulting trajectories were evaluated through simulation of the point spread function and slanted surface object suitable for modulation transfer function, contrast ratio, and SNR measurement. Results from the HEALPix view-order were compared to Generalized Spiral, Golden Means, and Random view-orders.

We report the first use of the HEALPix view-order to acquire in-vivo brain images.

Keywords: Magnetic resonance imaging (MRI), Deconvolution, Image enhancement/restoration(noise and artifact reduction)
Abstract: Spiral imaging has drawn increasing attention due to its high SNR efficiency and low sensitivity to motion. One major challenge in spiral imaging is the spatial blurring and distortion caused by off resonance and chemical shift. Previous joint deblurring and water-fat separation method has shown significant improvement over sequential methods, but could be very sensitive to phase errors introduced by inaccurate B0 or flow artefacts. This work presents a novel water-fat phase consistent constraint to improve the robustness of the joint deblurring method to such potential phase errors. In vivo experiments have been conducted to validate the feasibility of the proposed approach.

Keywords: Magnetic resonance imaging (MRI), Diffusion weighted imaging, Brain
Abstract: Magnetic resonance imaging (MRI) is a powerful tool to study the microstructure of the tissue. Two important modalities are diffusion MRI that provides information on the microstructural level and relaxometry that extracts the sensitivity of the signal to the biochemical environment by transverse and longitudinal relaxation times. We propose a modification of the signal model extending the classical apparent diffusion coefficient (ADC) into gamma distributed diffusion with kurtosis term. The estimation of the model parameters was done via a non-linear Levenberg-Marquardt method using a various number of diffusion-weighting gradients. The results show that the kurtosis model allows for a better interpolation compared to a more conventional model based on ADC.

Keywords: Image enhancement/restoration(noise and artifact reduction), Ultrasound, Magnetic resonance imaging (MRI)
Abstract: Presented here is a nonlinear apodization (NLA) method for processing magnetic resonance (MR) and ultrasound (US) images, which has been modified from its original use in processing radar imagery. This technique reduces Gibb’s artifacts (ringing) while preserving the boundary edges and the mainlobe width of the impulse response. This is done by selecting, pixel-by-pixel, the specific signal-domain windowing function (cosine-on-pedestal) that optimizes each point throughout the image. The windows are chosen from an infinite but bounded set, determined by weighting coefficients for the cosine-on-pedestal equation and the values of the pixels adjacent to the point of interest. By using this method, total sidelobe suppression is achievable without degrading the resolution of the mainlobe. In radar applications, this nonlinear apodization technique has shown to require fewer operations per pixel than other traditional apodization techniques. The preliminary results from applications on MR and US data are presented here.

Keywords: Nuclear imaging (e.g. PET, SPECT), Image reconstruction - analytical & iterative methods, Brain
Abstract: Attenuation compensation (AC) is a pre-requisite for reliable quantification and beneficial for visual interpretation tasks in SPECT imaging. Typical AC methods requires the presence of an attenuation map, which is obtained using a transmission scan, such as the CT scan. This has several disadvantages such as increased radiation dose, higher costs, and possible misalignment between SPECT and CT scans. Also, often a CT scan may be unavailable. In this context, we and others are showing that scattered photons in SPECT contain information to estimate the attenuation distribution. To exploit this observation, we propose a simple physics and learning-based method that uses the SPECT emission data in the photopeak and scatter windows to perform transmission-less ASC in SPECT. The method was developed in the context of quantitative 2-D dopamine-transporter (DaT)-scan SPECT imaging. The proposed method uses data acquired in the scatter window to reconstruct an initial estimate of the attenuation map using a physics-based approach. An auto-encoder is then trained to segment this initial estimate into soft tissue and bone regions. Pre-defined attenuation coefficients are assigned to these regions, yielding the reconstructed attenuation map. This is used to reconstruct the activity distribution using an ordered subsets expectation maximization (OSEM)-based reconstruction approach. We objectively evaluated the performance of this method using highly realistic simulation studies, where the task was to quantify activity uptake in the caudate and putamen regions of the brain from the reconstructed activity images. Normalized root mean square error (RMSE) between the true and estimated activity values were computed. We compared the performance of the proposed method to reconstructed images obtained using the true attenuation map, and to those obtained using the traditional Chang’s attenuation compensation approach. Our results showed no statistically significant differences between the activity estimates obtained using the proposed method and that with true attenuation map. Additionally, the proposed method significantly outperformed the Chang’s AC method. Overall, these results demonstrate excellent potential of the proposed method to perform transmission-less ASC and motivate further evaluation to other SPECT applications and using clinical data.

Keywords: Magnetic resonance imaging (MRI), Image reconstruction - analytical & iterative methods, Inverse methods
Abstract: We present a state estimation formulation for time-varying magnetic resonance imaging (MRI) utilizing data-driven a priori information. In state estimation, we utilize separate state evolution and observation models to model the time dependent image reconstruction problem. In this paper we compute the state estimates by using the Kalman filter (KF) and steady state Kalman smoother (KS). KF includes separate matrix for the process noise covariance that gives a priori information to the filter regarding the strength of the changes and the location in the image domain where the changes are expected to happen. In this work, we utilize a data-driven approach to compute this process noise covariance matrix, allowing for more accurate estimates in regions of activity and less noise in inactive regions. We construct this data-driven covariance matrix from the conventional sliding window (SW) estimates.

We evaluate the method by using golden angle sampled radial MRI, with both simulated and experimentally measured data. The simulated data corresponds to a functional MRI (fMRI) examination while the measured data is dynamic contrast-enhanced MRI measurement from a rat brain. In the KF estimates, only a single radial spoke of data was used to update the estimate at each time step. As such, a new estimate was formed after each new spoke, leading to faster frame rates when compared to the conventional sliding window. The results show that the state estimation approach with the data-driven process noise covariance can improve both spatial and temporal resolution.

Keywords: Surgical guidance/navigation, X-ray imaging
Abstract: In minimally invasive surgery, the clinician relies on image guidance to diagnose, plan and navigate. In order to update the live information during surgery, 2D X-ray images and pre-operatively acquired 3D CT images need to be registered. In this work, we propose a novel global double-view registration method based on the Branch and Bound algorithm to register them.

Keywords: Machine learning, Spine, Magnetic resonance imaging (MRI)
Abstract: 1.INTRODUCTION Individuals diagnosed with degenerative bone diseases such as osteoporosis are more susceptible to vertebral fractures which comprise almost 50% of all osteoporotic fractures in the United States per year. Thoracolumbar vertebral body fractures can be classified as wedge, biconcave, or crush, depending on the anterior (Ha), middle (Hm), and posterior (Hp) heights of each vertebral body. However, determining these height measurements in clinical workflow is time-consuming and resource intensive. Instance segmentation and keypoint detection network designs offer the ability to determine Ha, Hm, and Hp, in order to classify deformities according to the semi- or fully quantitative method. We investigated the accuracy of such algorithms for analyzing sagittal spine CT and MR images.

2.MATERIALS AND METHODS Sagittal spine MRI (998) and CT scans (35) were split into training and testing data. The training set was augmented (±15°rotation, ±30% contrast and brightness, random cropping) to a final size of 5667 vertebrae (1269 CT & 4398 MR). A testing set of 238 MR and 15 CT scans was used to evaluate the neural network. Mask RCNN (architecture for basic instance segmentation) was modified to include a 2D- UNet head (for better segmentation) along with a Keypoint RCNN network to detect 6 relevant vertebral keypoints (network design in Figure 1 w/one network per imaging modality). The effectiveness of the neural network was measured using two parameters: Dice score (ranges from 0 to 1 where 1=predicted overlay is manually segmented ground truth) and keypoint error distance (distance of predicted key-point location compared to manually labeled reference point).

3.RESULTS The neural network achieved an overall Dice coefficient of 0.968 and key-point error distance of 0.984 millimeters on the testing dataset. Mean percent error in Ha, Hm, and Hp height calculations (based on keypoints) was 0.13%.The neural network was able to process each scan slice with a mean time of 1.492 seconds.

4.CONCLUSIONS The neural network was able to determine morphometric measurements for detecting spinal fractures with high accuracy on sagittal MR and CT images. This approach could simplify the screening, detection of changes, and surgical planning in patients with vertebral deformities and fractures by reducing the burden on radiologists who have to do measurements manually.

Keywords: Machine learning, Brain, Magnetic resonance imaging (MRI)
Abstract: In this study, we used T-test loss for segmenting enlarged perivascular spaces (EPVS) area in the brain MRI. The segmentation model of EPVS is learned with T-test loss and combination of T1-weighted, T2-weighted, and the FLAIR brain MR image is used as input to effectively distinguish the EPVS. Our results showed that the cascaded U-Net with T-test loss (0.7002) showed the higher score than that with dice loss (0.6811) for segmentation of EVPS in the brain MRI.

Keywords: Nuclear imaging (e.g. PET, SPECT), Brain, Image segmentation

Keywords: Magnetic resonance imaging (MRI), Brain, Image synthesis
Abstract: Image synthesis in multi-contrast MRI can be achieved via an intensity transformation between source and target contrasts. Data-driven methods typically learn this transformation via L2 or L1 loss terms, reducing sensitivity to high spatial frequencies. To address this limitation, we proposed a novel approach based on conditional generative adversarial networks that synergistically combines adversarial, pixel-wise and perceptual losses.

Keywords: Microscopy - Light, Confocal, Fluorescence, Brain, Image segmentation

Keywords: Optical coherence tomography, Heart, Modeling - Anatomical, physiological and pathological
Abstract: Intravascular lithotripsy (IVL) is a plaque modification technique that delivers pressure waves to pre-treat heavily calcified vascular calcifications to aid successful stent deployment. IVL causes micro-fractures that can develop into macro-fractures which enable successful vessel expansion. Intravascular optical coherence tomography (IVOCT) has the penetration, resolution, and contrast to characterize coronary calcifications. We detected the presence of micro-fractures by comparing textures before and after IVL treatment (p = 0.0039). In addition, we used finite element model (FEM) to success-fully predict the location of macro-fracture. Results suggest that we can use our methods to understand and possibly clinically monitor IVL treatment.

Keywords: Microscopy - Light, Confocal, Fluorescence, Machine learning, Classification
Abstract: This study focuses on using deep image recognition to improve classification of microscopy images of cells. One of the main challenges of applying deep learning to microscopy data is that it is difficult to distinguish biological variations from technical noise. This is especially problematic in cellular imaging, where environmental and experimental conditions cause batch effects that are often more visually salient than the phenotypic variations of interest. We examine the Recursion Cellular Image Classification (CellSignal) Kaggle Challenge (Jul.-Sep., 2019). The evaluation metric of the challenge is the classification accuracy of small interfering RNA (siRNA), which corresponds to the classification accuracy of genetic perturbations. We introduce the concept of a “Control Delta Matrix,” which uses negative control images (images of cells with no siRNA exposure) to normalize input images. The input and control images are put through a deep convolutional pipeline to produce latent embeddings. Then, the embedding of the control is subtracted from that of the input to obtain the control delta matrix. Based on the intuition that the embedding of a negative control image is a representation of the batch’s inherent characteristics without siRNA influence, we hypothesize that its removal from the input image’s representation would isolate the siRNA’s effect and enable easier classification. Our preliminary results using the Control Delta Matrix in a DenseNet based model show steady improvement in both training accuracy (defined as siRNA classification accuracy) and training loss (defined as a weighted average of ArcFace and Softmax cross-entropy losses).

Keywords: Microscopy - Light, Confocal, Fluorescence, Image segmentation, Image registration
Abstract: The European Molecular Biology Laboratory (EMBL) is one of the world’s leading research institutions for the life sciences conducting multidisciplinary research to continuously advance our understanding of the fundamental mechanisms of life. The Ellenberg Group at EMBL performs state-of-the-art microscopy across scales and produces huge volume of data from different imaging modalities capturing various events inside living organisms, cells or organelles. Automated analysis of this data is important to quantify these events and connect the results from different experiments to understand how cells and their molecular machines are constructed and dynamically work together to carry out the essential functions of life. Automated analysis involves substantial development of sophisticated computational frameworks in order to deal with complexity and heterogeneity of image data and detect interesting events. It also enables scientists to observe the structures and dynamics of different sub-cellular components which are not possible to do experimentally. This presentation will address different fundamental biological questions to demonstrate that computational pipelines play key role to study them successfully. It will also demonstrate why image analysis frameworks are so important and how they make significant impact in achieving overall research goals in a multi-disciplinary research environment.

Keywords: Microscopy - Light, Confocal, Fluorescence, In-vivo cellular and molecular imaging, Animal models and imaging

Keywords: Machine learning, Diffusion weighted imaging, Brain
Abstract: This paper proposes a new deep learning model using relational reasoning with diffusion-weighted imaging (DWI) data. We investigate how effectively and comprehensively DWI tractography-based connectome predicts the impairment of expressive and receptive language ability in individual children with focal epilepsy (FE). The proposed model constitutes a combination of a dilated convolutional neural network (CNN) and a relation network (RN), with the latter being applied to the dependencies of axonal connections across cortical regions in the whole brain. The presented results from 51 FE children demonstrate that the proposed model outperforms other existing state-of-the-art algorithms to predict language abilities without depending on connectome densities, with average improvement of up to 96.2% and 83.8% in expressive and receptive language prediction, respectively.

Keywords: Probabilistic and statistical models & methods, Brain, Connectivity analysis
Abstract: We develop a graph node embedding Deep Neural Network that leverage on statistical outcome measure and graph structure given in the data. The objective is to identify regions of interests (ROIs) in the brain that are affected by topological changes of brain connectivity due to specific neurodegenerative diseases by enriching statistical group analysis. We tackle this problem by learning a latent space where statistical inference can be made more effectively. Our experiments on a large-scale Alzheimer's Disease dataset show promising result identifying ROIs that show statistically significant group differences separating even early and late Mild Cognitive Impairment (MCI) groups whose effect sizes are very subtle.

Keywords: Brain, Diffusion weighted imaging, Connectivity analysis
Abstract: Although diffusion tensor imaging (DTI) can identify white matter (WM) changes due to mild traumatic brain injury (mTBI), the task of within-subject longitudinal matching of DTI streamlines remains challenging in this condition. Here we combine (A) automatic, atlas-informed labeling of WM streamline clusters with (B) streamline prototyping and (C) Riemannian matching of elastic curves to quantify within-subject changes in WM structure properties, focusing on the arcuate fasciculus. The approach is demonstrated in a group of geriatric mTBI patients imaged acutely and ~6 months post-injury. Results highlight the utility of differen-tial geometry approaches when quantifying brain connectivity alterations due to mTBI.

Keywords: Connectivity analysis, Diffusion weighted imaging, Functional imaging (e.g. fMRI)
Abstract: Analysis of structural and functional connectivity of brain has become a fundamental approach in neuroscientific research. Despite several studies reporting consistent similarities as well as differences for structural and resting state (rs) functional connectomes, a comparative investigation of connectomic consistency between the two modalities is still lacking. Nonetheless, connectomic analysis comprising both connectivity types necessitate extra attention as consistency of connectivity differs across modalities, possibly affecting the interpretation of the results. In this study, we present a comprehensive analysis of consistency in structural and rs-functional connectomes obtained from longitudinal diffusion MRI and rs-fMRI data of a single healthy subject. We contrast consistency of deterministic and probabilistic tracking with that of full, positive, and negative functional connectivities across various connectome generation schemes, using correlation as a measure of consistency.

Keywords: Connectivity analysis, fMRI analysis
Abstract: The interactions among brain entities, commonly computed through pair-wise functional connectivity, are assumed to be manifestations of information processing which drive function. However, this focus on large-scale networks and their pair-wise temporal interactions is likely missing important information contained within fMRI data. We propose leveraging multi-connected features at both the voxel- and network-level to capture “multi-way entanglement” between networks and voxels, providing improved resolution of interconnected brain functional hierarchy. Entanglement refers to each brain network being heavily enmeshed with the activity of other networks. Under our multi-connectivity assumption, elements of a system simultaneously communicate and interact with each other through multiple pathways. As such we move beyond the typical pair-wise temporal partial or full correlation. We propose a framework to estimate functional multi-connectivity (FMC) by computing the relationship between system-wide connections of intrinsic connectivity networks (ICNs). Results show that FMC obtains information which is different from standard pair-wise analyses.

Keywords: Image segmentation, Machine learning, Diffusion weighted imaging
Abstract: A fundamental problem in brain imaging is the identification of volumes whose features distinguish two populations. One popular solution, Voxel-Based Analyses (VBA), glues together contiguous voxels with significant intra-voxel population differences. VBA's output regions may not be spatially consistent: each voxel may show a unique population effect. We introduce Agglomerative Region-Based Analysis (ARBA), which mitigates this issue to increase sensitivity. ARBA is an Agglomerative Clustering procedure, like Ward's method, which segments image sets in a common space to greedily maximize a likelihood function. The resulting regions are pared down to a set of disjoint regions that show statistically significant population differences via Permutation Testing. ARBA is shown to increase sensitivity over VBA in a detection task on multivariate Diffusion MRI brain images.

Keywords: Skin, Classification
Abstract: It is ideal to develop intelligent systems to accurately di- agnose diseases as human specialists. However, due to the highly imbalanced data problem between common and rare diseases, it is still an open problem for the systems to ef- fectively learn to recognize both common and rare diseases. We propose utilizing triplet modelling to overcome the data imbalance issue for the rare diseases. Moreover, we further develop a class-center based triplet loss in order to make the triplet-based learning more stable. Extensive evaluation on two skin image classification tasks shows that the triplet- based approach is very effective and outperforms the widely used methods for solving the imbalance problem, including oversampling, class weighting, and using focal loss.

Keywords: Machine learning, Pattern recognition and classification, Ultrasound
Abstract: Anatomical landmarks are a crucial prerequisite for many medical imaging tasks. Usually, the set of landmarks for a given task is predefined by experts. The landmark locations for a given image are then annotated manually or via machine learning methods trained on manual annotations. In this paper, in contrast, we present a method to automatically discover and localize anatomical landmarks in medical images. Specifically, we consider landmarks that attract the visual attention of humans, which we term visually salient landmarks. We illustrate the method for fetal neurosonographic images. First, full-length clinical fetal ultrasound scans are recorded with live sonographer gaze-tracking. Next, a convolutional neural network (CNN) is trained to predict the gaze point distribution (saliency map) of the sonographers on scan video frames. The CNN is then used to predict saliency maps of unseen fetal neurosonographic images, and the landmarks are extracted as the local maxima of these saliency maps. Finally, the landmarks are matched across images by clustering the landmark CNN features. We show that the discovered landmarks can be used within affine image registration, with average landmark alignment errors between 4.1% and 10.9% of the fetal head long axis length.

Keywords: Dimensionality reduction, Shape analysis, Machine learning
Abstract: In this paper, we propose a novel approach for manifold learning that combines the Earthmover's distance (EMD) with the diffusion maps method for dimensionality reduction. We demonstrate the potential benefits of this approach for learning shape spaces of proteins and other flexible macromolecules using a simulated dataset of 3-D density maps that mimic the non-uniform rotary motion of ATP synthase. Our results show that EMD-based diffusion maps require far fewer samples to recover the intrinsic geometry than the standard diffusion maps algorithm that is based on the Euclidean distance. To reduce the computational burden of calculating the EMD for all volume pairs, we employ a wavelet-based approximation to the EMD which reduces the computation of the pairwise EMDs to a computation of pairwise weighted-l1 distances between wavelet coefficient vectors.

Keywords: Pattern recognition and classification, Cervix, Graphical models & methods
Abstract: Cervical cancer is the second most prevalent cancer affecting women today. As the early detection of cervical carcinoma relies heavily upon screening and pre-clinical testing, digital cervicography has great potential as a primary or auxiliary screening tool, especially in low-resource regions due to its low cost and easy access. Although an automated cervical dysplasia detection system has been desirable, traditional fully-supervised training of such systems requires large amounts of annotated data which are often labor-intensive to collect. To alleviate the need for much manual annotation, we propose a novel graph convolutional network (GCN) based semi-supervised classification model that can be trained with fewer annotations. In existing GCNs, graphs are constructed with fixed features and can not be updated during the learning process. This limits their ability to exploit new features learned during graph convolution. In this paper, we propose a novel and more flexible GCN model with a feature encoder that adaptively updates the adjacency matrix during learning and demonstrate that this model design leads to improved performance. Our experimental results on a cervical dysplasia classification dataset show that the proposed framework outperforms previous methods under a semi-supervised setting, especially when the labeled samples are scarce.

Keywords: Magnetic resonance spectroscopy, Machine learning, Inverse methods
Abstract: This paper presents a novel method to reconstruct and separate metabolite and macromolecule (MM) signals in 1H magnetic resonance spectroscopic imaging (MRSI) data using learned nonlinear models. Specifically, deep autoencoder (DAE) networks were constructed and trained to learn the nonlinear low-dimensional manifolds, where the metabolite and MM signals reside individually. A regularized reconstruction formulation is proposed to integrate the learned models with signal encoding model to reconstruct and separate the metabolite and MM components. An efficient algorithm was developed to solve the associated optimization problem. The performance of the proposed method has been evaluated using simulation and experimental 1H-MRSI data. Efficient low-dimensional signal representation of the learned models and improved metabolite/MM separation over the standard parametric fitting based approach have been demonstrated.

Keywords: Machine learning, Spine
Abstract: In this paper we introduce the Ladder Algorithm; a novel recurrent algorithm to detect repetitive structures in natural images with high accuracy using little training data.

We then demonstrate the algorithm on the task of extracting vertebrae from whole spine magnetic resonance scans with only lumbar MR scans for training data.

It is shown to achieve high performance with 99.8% precision and recall, exceeding current state of the art approaches for lumbar vertebrae detection in T1 and T2 weighted scans. It also generalises to whole spine images with minimal drop in accuracy, achieving a detection rate of 99.4%.

Keywords: Machine learning, Ultrasound, Heart
Abstract: Accurate interpretation and analysis of echocardiography is important in assessing cardiovascular health. However, motion tracking often relies on accurate segmentation of the myocardium, which can be difficult to obtain due to inherent ultrasound properties. In order to address this limitation, we propose a semi-supervised joint learning network that exploits overlapping features in motion tracking and segmentation. The network simultaneously trains two branches: one for motion tracking and one for segmentation. Each branch learns to extract features relevant to their respective tasks and shares them with the other. Learned motion estimations propagate a manually segmented mask through time, which is used to guide future segmentation predictions. Physiological constraints are introduced to enforce realistic cardiac behavior. Experimental results on synthetic and in vivo canine 2D+t echocardiographic sequences outperform some competing methods in both tasks.

Keywords: Image segmentation, Machine learning, Heart
Abstract: Automated segmentation of aortic valve components using pre-operative CT scans would help provide quantitative metrics for better treatment planning of valve replacement procedures and create inputs for simulations such as finite element analysis. U-net has been used extensively for segmentation in medical imaging, but naive application of this model onto large 3D images leads to memory issues and drop in accuracy. Hence, we propose an architecture sequentially combining it with a Spatial Transformer Network (STN), which effectively transforms the original image to a consistent subregion containing the aortic valve. The addition of STN improves segmentation performance while significantly decreasing training time. Training is performed end-to-end, with no additional supervision for the STN. This framework may be useful in other medical imaging applications where the entity of interest is sparse, has a fixed number of instances, and exhibits shape regularity.

Keywords: Image segmentation, Heart, Computed tomography (CT)
Abstract: Providing closed and well-connected boundaries of coronary artery is essential to assist cardiologists in the diagnosis of coronary artery disease (CAD). Recently, several deep learning-based methods have been proposed for boundary detection and segmentation in a medical image. However, when applied to coronary wall detection, they tend to produce disconnected and inaccurate boundaries. In this paper, we propose a novel boundary detection method for coronary arteries that focuses on the continuity and connectivity of the boundaries. In order to model the spatial continuity of consecutive images, our hybrid architecture takes a volume (i.e., a segment of the coronary artery) as input and detects the boundary of the target slice (i.e., the central slice of the segment). Then, to ensure closed boundaries, we propose a contour-constrained weighted Hausdorff distance loss. We evaluate our method on a dataset of 34 patients of coronary CT angiography scans with curved planar reconstruction (CCTA-CPR) of the arteries (i.e., cross-sections). Experiment results show that our method can produce smooth closed boundaries outperforming the state-of-the-art accuracy.

Keywords: Magnetic resonance imaging (MRI), Image reconstruction - analytical & iterative methods, Machine learning
Abstract: Cardiac magnetic resonance imaging (CMR) is a noninvasive imaging modality that provides a comprehensive evaluation of the cardiovascular system. The clinical utility of CMR is hampered by long acquisition times, however. In this work, we propose and validate a plug-and-play (PnP) method for CMR reconstruction from undersampled multi-coil data. To fully exploit the rich image structure inherent in CMR, we pair the PnP framework with a deep learning (DL)-based denoiser that is trained using spatiotemporal patches from high-quality, breath-held cardiac cine images. The resulting "PnP-DL" method iterates over data consistency and denoising subroutines. We compare the reconstruction performance of PnP-DL to that of compressed sensing (CS) using eight breath-held and ten real-time (RT) free-breathing cardiac cine datasets. We find that, for breath-held datasets, PnP-DL offers more than one dB advantage over commonly used CS methods. For RT free-breathing datasets, where ground truth is not available, PnP-DL receives higher scores in qualitative evaluation. The results highlight the potential of PnP-DL to accelerate RT CMR.

Keywords: Image segmentation, Machine learning, Magnetic resonance imaging (MRI)
Abstract: Deep convolutional neural networks have been applied to medical image segmentation tasks successfully in recent years by taking advantage of a large amount of training data with golden standard annotations. However, it is difficult and expensive to obtain good-quality annotations in practice. This work aims to propose a novel semi-supervised learning framework to improve the ventricle segmentation from 2D cine MR images. Our method is efficient and effective by computing soft labels dynamically for the unlabeled data. Specifically, we obtain the soft labels, rather than hard labels, from a teacher model in every learning iteration. The uncertainty of the target label of unlabeled data is intrinsically encoded in the soft label. The soft label can be improved towards the ideal target in training. We use a separate loss to regularize the unlabeled data to produce similar probability distribution as the soft labels in each iteration. Experiments show that our method outperforms a state-of-the-art semi-supervised method.

Keywords: Magnetic resonance imaging (MRI), Image segmentation, Machine learning
Abstract: This study presents a novel automated algorithm to segment the left atrium (LA) from 2, 3 and 4-chamber long-axis cardiac cine magnetic resonance image (MRI) sequences using deep convolutional neural network (CNN). The objective of the segmentation process is to delineate the boundary between myocardium and endocardium and exclude the mitral valve so that the results could be used for generating clinical measurements such as strain and strain rate. As such, the segmentation needs to be performed using open contours, a more challenging problem than the pixel-wise semantic segmentation performed by existing machine learning-based approaches such as U-net. The proposed neural net is built based on pre-trained CNN Inception V4 architecture, and it predicts a compressed vector by applying a multi-layer autoencoder, which is then back-projected into the segmentation contour of the LA to perform the delineation using open contours. Quantitative evaluations were performed to compare the performances of the proposed method and the current state-of-the-art U-net method. Both methods were trained using 6195 images acquired from 80 patients and evaluated using 1515 images acquired from 20 patients where the datasets were obtained retrospectively, and ground truth manual segmentation was provided by an expert radiologist. The proposed method yielded an average Dice score of 93.1 % and Hausdorff distance of 4.2 mm, whereas the U-net yielded 91.6 % and 11.9 mm for Dice score and Hausdorff distance metrics, respectively. The quantitative evaluations demonstrated that the proposed method performed significantly better than U-net in terms of Hausdorff distance in addition to providing open contour-based segmentation for the LA.

Keywords: Ultrasound, Fetus, Image synthesis
Abstract: Synthesis of anatomically realistic ultrasound images could be potentially valuable in sonographer training and to provide training images for algorithms, but is a challenging technical problem. Generating examples where different image attributes can be controlled may also be useful for tasks such as semi-supervised classification and regression to augment costly human annotation. In this paper, we propose using an information maximizing generative adversarial network with a least-squares loss function to generate new examples of fetal brain ultrasound images from clinically acquired healthy subject twenty-week anatomy scans. The unsupervised network succeeds in disentangling natural clinical variations in anatomical visibility and image acquisition parameters, which allows for user-control in image generation. To evaluate our method, we also introduce an additional synthetic fetal ultrasound specific image quality metric called the Frechet SonoNet Distance (FSD) to quantitatively evaluate synthesis quality. To the best of our knowledge, this is the first work that generates ultrasound images with a generator network trained on clinical acquisitions where governing parameters can be controlled in a visually interpretable manner.

Keywords: Image synthesis, Magnetic resonance imaging (MRI)
Abstract: The one-to-one mapping is necessary for MRI image synthesis as MRI images are unique to the patient. State-of-the-art approaches for image synthesis from domain X to domain Y learn a convolutional neural network that meticulously maps between the domains. A different network is typically implemented to map along the opposite direction, from Y to X. In this paper, we explore the possibility of only wielding one network for bi-directional image synthesis. In other words, such an autonomous learning network implements a self-inverse function. A self-inverse network shares several distinct advantages: only one network instead of two, better generalization and more restricted parameter space. Most importantly, a self-inverse function guarantees a one-to-one mapping, a property that cannot be guaranteed by earlier approaches that are not self-inverse. The experiments on MRI T1 and T2 images show that, compared with the baseline approaches that use two separate models for the image synthesis along with two directions, our self-inverse network achieves better synthesis results in terms of standard metrics. Finally, our sensitivity analysis confirms the feasibility of learning a one-to-one mapping function for MRI image synthesis.

Keywords: Histopathology imaging (e.g. whole slide imaging), Breast, Image synthesis
Abstract: We propose a virtual staining methodology based on Generative Adversarial Networks to map histopathology images of breast cancer tissue from H&E stain to PHH3 and vice versa. We use the resulting synthetic images to build Convolutional Neural Networks (CNN) for automatic detection of mitotic figures, a strong prognostic biomarker used in routine breast cancer diagnosis and grading. We propose several scenarios, in which CNN trained with synthetically generated histopathology images perform on par with or even better than the same baseline model trained with real images. We discuss the potential of this application to scale the number of training samples without the need for manual annotations.

Keywords: Microscopy - Light, Confocal, Fluorescence, Image synthesis, Machine learning
Abstract: The automatic reconstruction of single neuron cells from microscopic images is essential to enabling large-scale data-driven investigations in neuron morphology research. However, the performances of single neuron reconstruction algorithms are constrained by both the quantity and the quality of the annotated 3D microscopic images since the annotating single neuron models is highly labour intensive. We propose a framework for synthesizing microscopy-realistic 3D neuron images from simulated single neuron skeletons using conditional Generative Adversarial Networks (cGAN). We build the generator network with multi-resolution sub-modules to improve the output fidelity. We evaluate our framework on Janelia-Fly dataset from the BigNeuron project. With both qualitative and quantitative analysis, we show that the proposed framework outperforms the other state-of-the-art methods regarding the quality of the synthetic neuron images. We also show that combining the real neuron images and the synthetic images generated from our framework can improve the performance of neuron segmentation.

Keywords: Image synthesis, Machine learning, Ultrasound
Abstract: Ultrasound imaging relies on sensing of waves returned after interaction with scattering media present in biological tissues. An acoustic pulse transmitted by a single element transducer dilates along the direction of propagation, and is observed as 1D point spread function (PSF) in A-mode imaging. In 2D B-mode imaging, a 1D array of transducer elements is used and dilation of pulse is also observed along the direction of these elements, manifesting a 2D PSF. In 3D B-mode imaging using a 2D matrix of transducer elements, a 3D PSF is observed. Fast simulation of a 3D B-mode volume by way of convolutional transformer networks to learn the PSF family would require a training dataset of true 3D volumes which are not readily available. Here we start in Stage 0 with a simple physics based simulator in 3D to generate speckles from a tissue echogenicity map. Next in Stage 1, we learn a multilinear separable 2D convolutional neural network using 1D convolutions to model PSF family along direction of ultrasound propagation and orthogonal to it. This is adversarially trained using a visual Turing test on 2D ultrasound images. The PSF being circularly symmetric about an axis parallel to the direction of wave propagation, we simulate full 3D volume, by way of alternating the direction of 1D convolution along 2 axes that are mutually orthogonal to the direction of wave propagation. We validate performance using visual Turing test with experts and distribution similarity measures.

Keywords: Spine, Image synthesis, Machine learning
Abstract: In this paper, we introduce a fully unsupervised approach for the synthesis of CT images of the lumbar spine, used for image-guided surgical procedures, from a T2-weighted MRI acquired for diagnostic purposes. Our approach makes use of a trainable pre-processing pipeline using a low-capacity fully convolutional network, to normalize the input MRI data, in cascade with FC-ResNets, to segment the vertebral bodies and pedicles. A pseudo-3D Cycle GAN architecture is proposed to include neighboring slices in the synthesis process, along with a cyclic loss function ensuring consistency between MRI and CT synthesis. Clinical experiments were performed on the SpineWeb dataset, totalling 18 patients with both MRI and CT. Quantitative comparison to expert CT segmentations yields an average Dice score of 83 +/- 1.6 on synthetic CTs, while a comparison to CT annotations yielded a landmark localization error of 2.2 +/- 1.4mm. Intensity distributions and mean absolute errors in Hounsfield units also show promising results, illustrating the strong potential and versatility of the pipeline by achieving clinically viable CT scans which can be used for surgical guidance.

Keywords: Classification, Retinal imaging, Image synthesis
Abstract: Due to new eye diseases discovered every year, doctors may encounter some rare or unknown diseases. Similarly, in medical image recognition field, many practical medical classification tasks may encounter the case where some testing samples belong to some rare or unknown classes that have never been observed or included in the training set, which is termed as an open-set problem. As rare diseases samples are difficult to be obtained and included in the training set, it is reasonable to design an algorithm that recognizes both known and unknown diseases. Towards this end, this paper leverages a novel generative adversarial network (GAN) based synthetic learning for open-set retinal optical coherence tomography (OCT) image recognition. Specifically, we first train an auto-encoder GAN and a classifier to reconstruct and classify the observed images, respectively. Then a subspace-constrained synthesis loss is introduced to generate images that locate near the boundaries of the subspace of images corresponding to each observed disease, meanwhile, these images cannot be classified by the pre-trained classifier. In other words, these synthesized images are categorized into an unknown class. In this way, we can generate images belonging to the unknown class, and add them into the original dataset to retrain the classifier for the unknown disease discovery

Keywords: Image synthesis, Ultrasound
Abstract: Ultrasound (US) is widely accepted in clinic for anatomical structure inspection. However, lacking in resources to practice US scan, novices often struggle to learn the operation skills. Also, in the deep learning era, automated US image analysis is limited by the lack of annotated samples. Efficiently synthesizing realistic, editable and high resolution US images can solve the problems. The task is challenging and previous methods can only partially complete it. In this paper, we devise a new framework for US image synthesis. Particularly, we firstly adopt a Sgan to introduce background sketch upon object mask in a conditioned generative adversarial network. With enriched sketch cues, Sgan can generate realistic US images with editable and fine-grained structure details. Although effective, Sgan is hard to generate high resolution US images. To achieve this, we further implant the Sgan into a progressive growing scheme (PGSgan). By smoothly growing both generator and discriminator, PGSgan can gradually synthesize US images from low to high resolution. By synthesizing ovary and follicle US images, our extensive perceptual evaluation, user study and segmentation results prove the promising efficacy and efficiency of the proposed PGSgan.

Keywords: Image synthesis, Skin, Image segmentation
Abstract: Data synthesis is an important tool for improving data availability in cases where data is hard to capture or annotate. In the context of skin lesions data, data synthesis has been used for data augmentation in automated classification methods or for supporting training of dermoscopic images visual inspection. In this paper, we propose a simple yet effective approach for diverse skin lesion image synthesis using conditional generative adversarial networks. Our pipeline takes as input a random Bezier curve representing the lesion mask, and two texture patches: one for skin, and one for lesion; and synthesizes a new dermoscopic image. Our method generates images where lesions and skin reproduce the corresponding provided texture patches, and the lesion conforms to the provided Bezier mask. Our results report realistic controllable synthesis and improved performance for skin lesion segmentation task considering different semantic segmentation networks in a public challenge in comparison to classic data augmentation.

Keywords: Image synthesis, Magnetic resonance imaging (MRI), Brain
Abstract: Magnetic Resonance (MR) images of different modalities can provide complementary information for clinical diagnosis, but whole modalities are often costly to access. Most existing methods only focus on synthesizing missing images between two modalities, which limits their robustness and efficiency when multiple modalities are missing. To address this problem, we propose a multi-modality generative adversarial network (MGAN) to synthesize three high-quality MR modalities (FLAIR, T1 and T1ce) from one MR modality T2 simultaneously. The experimental results show that the quality of the synthesized images by our proposed methods is better than the one synthesized by the baseline model, pix2pix. Besides, for MR brain image synthesis, it is important to preserve the critical tumor information in the generated modalities, so we further introduce a multi-modality tumor consistency loss to MGAN, called TC-MGAN. We use the synthesized modalities by TC-MGAN to boost the tumor segmentation accuracy, and the results demonstrate its effectiveness.

Keywords: Ultrasound, Liver, Image synthesis
Abstract: External beam radiation therapy (EBRT) is a therapeutic modality often used for the treatment of various types of cancer. EBRT’s efficiency highly depends on accurate tracking of the target to be treated and therefore requires the use of real-time imaging modalities such as ultrasound (US) during treatment. While US is cost effective and non-ionizing, 2D US is not well suited to track targets that displace in 3D, while 3D US is challenging to integrate in real-time due to insufficient temporal frequency. In this work, we present a 3D inference model based on fully convolutional networks combined with a spatial transformative network (STN) layer, which given a 2D US image and a baseline 3D US volume as inputs, can predict the deformation of the baseline volume to generate an up-to-date 3D US volume in real-time. We train our model using 20 4D liver US sequences taken from the CLUST15 3D tracking challenge, testing the model on image tracking sequences. The proposed model achieves a normalized cross-correlation of 0.56 in an ablation study and a mean landmark location error of 2.92 ± 1.67mm for target anatomy tracking. These promising results demonstrate the potential of generative STN models for predicting 3D motion fields during EBRT.

Keywords: Elastography imaging, Viscoelasticity imaging, Inverse methods
Abstract: Elastography refers to mapping mechanical properties in a material based on measuring wave motion in it using noninvasive optical, acoustic or magnetic resonance imaging methods. For example, increased stiffness will increase wavelength. Stiffness and viscosity can depend on both location and direction. A material with aligned fibers or layers may have different stiffness and viscosity values along the fibers or layers versus across them. Converting wave measurements into a mechanical property map or image is known as reconstruction. To make the reconstruction problem analytically tractable, isotropy and homogeneity are often assumed, and the effects of finite boundaries are ignored. But, infinite isotropic homogeneity is not the situation in most cases of interest, when there are pathological conditions, material faults or hidden anomalies that are not uniformly distributed in fibrous or layered structures of finite dimension. Introduction of anisotropy, inhomogeneity and finite boundaries complicates the analysis forcing the abandonment of analytically-driven strategies, in favor of numerical approximations that may be computationally expensive and yield less physical insight. A new strategy, Transformation Elastography (TE), is proposed that involves spatial distortion in order to make an anisotropic problem become isotropic. The fundamental underpinnings of TE have been proven in forward simulation problems. In the present paper a TE approach to inversion and reconstruction is introduced and validated based on numerical finite element simulations.

Keywords: Perfusion imaging, Ultrasound, Image segmentation
Abstract: Contrast-enhanced ultrasound (CEUS) has been a popular clinical imaging technique for the dynamic visualization of the tumor microvasculature. Due to the heterogeneous intratumor vessel distribution and ambiguous lesion boundary, automatic tumor segmentation in the CEUS sequence is challenging. To overcome these difficulties, we propose a novel network, CEUS-Net, which is a novel U-net network infused with our designed feature-reweighted dense blocks. Specifically, CEUS-Net incorporates the dynamic channel-wise feature re-weighting into the Dense block for adapting the importance of learned lesion-relevant features. Besides, in order to efficiently utilize dynamic characteristics of CEUS modality, our model attempts to learn spatial-temporal features encoded in diverse enhancement patterns using a multichannel convolutional module. The CEUS-Net has been tested on tumor segmentation tasks of CEUS images from breast and thyroid lesions. It results in the dice index of 0.84, and 0.78 for CEUS segmentation of breast and thyroid respectively.

Keywords: Ultrasound, Tissue, Classification
Abstract: H-scan imaging is a new ultrasound (US) technique used to visualize the relative size of acoustic scatterers. The purpose of this study was to evaluate the sensitivity of H-scan US imaging to scatterer size and comparison to histological sections of tumor tissue. Image data was acquired using a programmable US scanner (Vantage 256, Verasonics Inc) equipped with a 256-element L22-8v capacitive micromachined ultrasonic transducer (CMUT, Kolo Medical). To generate the H-scan US image, three parallel convolution filters were applied to the radiofrequency (RF) data sequences to measure the relative strength of the backscattered US signals. H-scan US imaging was used to image a gelatin-based heterogenous phantom and breast tumor-bearing mice (N = 4). Excised tumor tissue underwent histologic processing and the cells were segmented to compute physical size measurements at the cellular level followed by spatial correlation with H-scan US image features. The in vitro results show that there was an improvement in the contrast-to-noise ratio (CNR) of 44.1% for H-scan compared to B-scan US imaging. Preliminary animal studies revealed there was a statistically significant relationship between H-scan US and physical size measures at the cell level (R2 > 0.95, p < 0.02). Overall, this study details the first experimental evidence that H-scan US image findings are directly related to physical cell size of the underlying bulk tumor tissue.

Keywords: Machine learning, Image segmentation, Ultrasound
Abstract: Deep Neural Networks (DNNs) suffer from the performance degradation when image appearance shift occurs, especially in ultrasound (US) image segmentation. In this paper, we propose a novel and intuitive framework to remove the appearance shift, and hence improve the generalization ability of DNNs. Our work has three highlights. First, we follow the spirit of universal style transfer to remove appearance shifts, which was not explored before for US images. Without sacrificing image structure details, it enables the arbitrary style-content transfer. Second, accelerated with Adaptive Instance Normalization block, our framework achieved real-time speed required in the clinical US scanning. Third, an efficient and effective style image selection strategy is proposed to ensure the target-style US image and testing content US image properly match each other. Experiments on two large US datasets demonstrate that our methods are superior to state-of-the-art methods on making DNNs robust against various appearance shifts.

Keywords: Ultrasound, Fetus, Machine learning
Abstract: 3D ultrasound (US) can improve the prenatal examinations for fetal growth monitoring. Detecting anatomical landmarks of fetus in 3D US has plenty of applications. Classical methods directly regress the coordinates or gaussian heatmaps of landmarks. However, these methods tend to show drawbacks when facing with the large volume and poor image quality of 3D US images. Different from previous methodology, in this work, we propose a successful and first investigation about exploiting object detection framework for landmark detection in 3D US. By regressing multiple parameters of the landmark-centered bounding box (B-box) with strict criteria, object detection framework presents potentials in outperforming previous landmark detection methods. Specifically, we choose the region proposal network (RPN) with localization and classification branches as our backbone for detection efficiency. Based on 3D RPN, we propose to adopt an IoU-balance loss to enhance the communication between two branches and promote the landmark localization. Furthermore, we build a distance based graph prior to regularize the landmark localization and therefore reduce false positives. We validate our method on the challenging task of detection for five fetal facial landmarks. Regarding the landmark localization and classification criteria, our method outperforms the state-of-the-art methods in efficacy and efficiency.

Keywords: Ultrasound, Machine learning, Image segmentation
Abstract: Ultrasound (US) is one of the most commonly used imaging modalities in both diagnosis and surgical interventions due to its low-cost, safety, and non-invasive characteristic. US image segmentation is currently a unique challenge because of the presence of speckle noise. As manual segmentation requires considerable efforts and time, the development of automatic segmentation algorithms has attracted researchers’ attention. Although recent methodologies based on convolutional neural networks have shown promising performances, their success relies on the availability of a large number of training data, which is prohibitively difficult for many applications. There- fore, in this study we propose the use of simulated US images and natural images as auxiliary datasets in order to pre-train our segmentation network, and then to fine-tune with limited in vivo data. We show that with as little as 19 in vivo images, fine-tuning the pre-trained network improves the dice score by 21% compared to training from scratch. We also demonstrate that if the same number of natural and simulation US images is available, pre-training on simulation data is preferable.

Keywords: Ultrasound
Abstract: Three-dimensional (3D) H-scan ultrasound (US) is a new high-resolution imaging technology for voxel-level tissue classification. For the purpose of validation, a simulated H-scan US imaging system was developed to comprehensively study the sensitivity to scatterer size in volume space. A programmable research US system (Vantage 256, Verasonics Inc, Kirkland, WA) equipped with a custom volumetric imaging transducer (4DL7, Vermon, Tours, France) was used for US data acquisition and comparison to simulated findings. Preliminary studies were conducted using homogeneous phantoms embedded with acoustic scatterers of varying sizes (15, 30, 40 or 250 μm). Both simulation and experimental results indicate that the H-scan US imaging method is more sensitive than B-mode US in differentiating US scatterers of varying size. Overall, this study proved useful for evaluating H-scan US imaging of tissue scatterer patterns and will inform future technology research and development.

Keywords: Shape analysis, Ultrasound, Modeling - Anatomical, physiological and pathological
Abstract: Deformable medial modeling is an approach to extracting clinically useful features of the morphological skeleton of anatomical structures in medical images. Similar to any deformable modeling technique, it requires a pre-defined model, or synthetic skeleton, of a class of shapes before modeling new instances of that class. The creation of synthetic skeletons often requires manual interaction, and the deformation of the synthetic skeleton to new target geometries is prone to registration errors if not well initialized. This work presents a fully automated method for creating synthetic skeletons (i.e., 3D boundary meshes with medial links) for flat, oblong shapes that are homeomorphic to a sphere. The method rotationally cross-sections the 3D shape, approximates a 2D medial model in each cross-section, and then defines edges between nodes of neighboring slices to create a regularly sampled 3D boundary mesh. In this study, we demonstrate the method on 62 segmentations of placentas in first-trimester 3D ultrasound images and evaluate its compatibility and representational accuracy with an existing deformable modeling method. The method may lead to extraction of new clinically meaningful features of placenta geometry, as well as facilitate other applications of deformable medial modeling in medical image analysis.

Keywords: Ultrasound, Fetus, Machine learning
Abstract: Recent advances in deep learning have achieved promising performance for medical image analysis, while in most cases ground-truth annotations from human experts are necessary to train the deep model. In practice, such annotations are expensive to collect and can be scarce for medical imaging applications. Therefore, there is significant interest in learning representations from unlabelled raw data. In this paper, we propose a self-supervised learning approach to learn meaningful and transferable representations from medical imaging video without any type of human annotation. We assume that in order to learn such a representation, the model should identify anatomical structures from the unlabelled data. Therefore we force the model to address anatomy-aware tasks with free supervision from the data itself. Specifically, the model is designed to correct the order of a reshuffled video clip and at the same time predict the geometric transformation applied to the video clip. Experiments on fetal ultrasound video show that the proposed approach can effectively learn meaningful and strong representations, which transfer well to downstream tasks like standard plane detection and saliency prediction.

Keywords: X-ray imaging, Lung, Machine learning
Abstract: We propose a novel deep neural network architecture for normalcy detection in chest x-ray images. This architecture treats the problem as fine-grained binary classification in which the normal cases are well-defined as a class while leaving all other cases in the broad class of abnormal. It employs several components that allow generalization and prevent overfitting across demographics. The model is trained and validated on a large public dataset of frontal chest X-ray images. It is then tested independently on images from a clinical institution of differing patient demographics using a three radiologist consensus for ground truth labeling. The model provides an area under ROC curve of 0.96 when tested on 1271 images. Using this model, we can automatically remove nearly a third of disease-free chest X-ray screening images from the workflow, without introducing any false negatives thus raising the potential of expediting radiology workflows in hospitals in future.

Keywords: Computed tomography (CT), Lung, Machine learning
Abstract: Registration of lungs in thoracic computed tomography (CT) images produces a dense correspondence which can be analyzed to estimate local tissue expansion. However, the validity of this local expansion estimate is dependent on the accuracy of the image registration. In this work, a convolutional neural network (CNN) model is used to directly estimate the local tissue expansion between lungs imaged at two lung volumes, without requiring image registration. The network was trained with 5705 subjects from COPDGene with varying degrees of disease severity. The CNN-based model was evaluated with 3046 subjects from COPDGene. At the global scale, the mean lung expansion estimated from the CNN-based and registration-based models were highly correlated (rs = 0.945). At the local scale, the proposed method achieved a voxelwise Spearman correlation of 0.871 ± 0.080. At the regional scale, Dice coefficient for high and low func- tioning regions was 0.806 ± 0.065 and 0.805 ± 0.066, respectively. The results indicate the CNN-based model was able to reproduce image registration derived tissue expansion images without explicitly estimating the correspondence.

Keywords: Computed tomography (CT), Lung, Machine learning
Abstract: Current lung nodule detection methods generate several candidate regions per nodule, such that a Non-Maximum Suppression (NMS) algorithm is required to select a single region per nodule while eliminating the redundant ones. GossipNet is a 1D Neural Network (NN) for NMS, which can learn the NMS parameters rather than relying on handcrafted ones. However, GossipNet does not take advantage of image features to learn NMS. We use Faster R-CNN with ResNet18 for candidate region detection and present FeatureNMS --- a neural network that provides additional image features to the input of GossipNet, which result from a transformation over the voxel intensities of each candidate region in the CT image. Experiments indicate that FeatureNMS can improve nodule detection in 2.33% and 0.91%, on average, when compared to traditional NMS and the original GossipNet, respectively.

Keywords: Lung, Computer-aided detection and diagnosis (CAD), Computed tomography (CT)
Abstract: Pulmonary nodule detection plays an important role in lung cancer screening with low-dose computed tomography (CT) scans. It remains challenging to build nodule detection deep learning models with good generalization performance due to unbalanced positive and negative samples. In order to overcome this problem and further improve state-of-the-art nodule detection methods, we develop a novel deep 3D convolutional neural network with an Encoder-Decoder structure in conjunction with a region proposal network. Particularly, we utilize a dynamically scaled cross entropy loss to reduce the false positive rate and combat the sample imbalance problem associated with nodule detection. We adopt the squeeze-and-excitation structure to learn effective image features and utilize inter-dependency information of different feature maps. We have validated our method based on publicly available CT scans with manually labelled ground-truth obtained from LIDC/IDRI dataset and its subset LUNA16 with thinner slices. Ablation studies and experimental results have demonstrated that our method could outperform state-of-the-art nodule detection methods by a large margin.

Keywords: Lung, Classification, Computed tomography (CT)
Abstract: While several datasets containing CT images of lung nodules exist, they do not contain definitive diagnoses and often rely on radiologists' visual assessment for malignancy rating. This is in spite of the fact that lung cancer is one of the top three most frequently misdiagnosed diseases based on visual assessment. In this paper, we propose a dataset of difficult-to-diagnose lung nodules based on data from the National Lung Screening Trial (NLST), which we refer to as NLSTx. In NLSTx, each malignant nodule has a definitive ground truth label from biopsy. Herein, we also propose a novel deep convolutional neural network (CNN) / recurrent neural network framework that allows for use of pre-trained 2-D convolutional feature extractors, similar to those developed in the ImageNet challenge. Our results show that the proposed framework achieves comparable performance to an equivalent 3-D CNN while requiring half the number of parameters.

Keywords: Machine learning, Computer-aided detection and diagnosis (CAD)
Abstract: Diagnosis and treatment of multiple pulmonary nodules are clinically important but challenging. Prior studies on nodule characterization use solitary-nodule approaches on multiple nodular patients, which ignores the relations between nodules. In this study, we propose a multiple instance learning (MIL) approach and empirically prove the benefit to learn the relations between multiple nodules. By treating the multiple nodules from a same patient as a whole, critical relational information between solitary-nodule voxels is extracted. To our knowledge, it is the first study to learn the relations between multiple pulmonary nodules. Inspired by recent advances in natural language processing (NLP) domain, we introduce a self-attention transformer equipped with 3D CNN, named NoduleSAT, to replace typical pooling-based aggregation in multiple instance learning. Extensive experiments on lung nodule false positive reduction on LUNA16 database, and malignancy classification on LIDC-IDRI database, validate the effectiveness of the proposed method.

Keywords: X-ray imaging, Pattern recognition and classification, Lung
Abstract: The automatic detection of critical findings in chest X-rays (CXR), such as pneumothorax, is important for assisting radiologists in their clinical workflow like triaging time-sensitive cases and screening for incidental findings. While deep learning (DL) models has become a promising predictive technology with near-human accuracy, they commonly suffer from a lack of explainability, which is an important aspect for clinical deployment of DL models in the highly regulated healthcare industry. For example, localizing critical findings in an image is useful for explaining the predictions of DL classification algorithms. While there have been a host of joint classification and localization methods for computer vision, the state-of-the-art DL models require locally annotated training data in the form of pixel level labels or bounding box coordinates. In the medical domain, this requires an expensive amount of manual annotation by medical experts for each critical finding. This requirement becomes a major barrier for training models that can rapidly scale to various findings. In this work, we address these shortcomings with an interpretable DL algorithm based on multi-instance learning that jointly classifies and localizes critical findings in CXR without the need for local annotations. We show competitive classification results on three different critical findings (pneumothorax, pneumonia, and pulmonary edema) from three different CXR datasets.

Keywords: Lung, Computed tomography (CT), Quantification and estimation
Abstract: Quantitative computed tomography (CT)-based characterization of bronchial metrics is increasingly being used to investigate chronic obstructive pulmonary disease (COPD)-related phenotypes. Automated methods for airway measurements benefit large multi-site studies by reducing cost and subjectivity errors. Critical challenges for CT-based analysis of airway morphology are related to location of lumen and wall transitions in the presence of varying scales and intensity-contrasts from proximal to distal sites. This paper introduces locally adaptive half-max methods to locate airway lumen and wall transitions and compute cross-sectional lumen area and wall-thickness. Also, the method uses a consistency analysis of wall-thickness to avoid adjoining-structure-artifacts. Experimental results show that computed bronchial measures at individual anatomic airway tree locations are repeat CT scan reproducible with intra-class correlation coefficient (ICC) values exceeding 0.9 and 0.8 for lumen-area and wall-thickness, respectively. Observed ICC values for derived morphologic measures, e.g., lumen-area compactness (ICC>0.67) and tapering (ICC>0.47) are relatively lower.

Keywords: Magnetic resonance imaging (MRI), Image segmentation
Abstract: We develop a fully automated airway segmentation method to segment the vocal tract airway from surrounding soft tissue in speech MRI. We train a U-net architecture to learn the end to end mapping between a mid-sagittal image (at the input), and the manually segmented airway (at the output). We base our training on the open source University of Southern California’s (USC) speech morphology MRI database consisting of speakers producing a variety of sustained vowel and consonant sounds. Once trained, our model performs fast airway segmentations on unseen images at the order of 210 ms/slice on a modern CPU with 12 cores. Using manual segmentation as a reference, we evaluate the performances of the proposed U-net airway segmentation, against existing seed-growing segmentation, and manual segmentation from a different user. We demonstrate improved DICE similarity with U-net compared to seed-growing, and minor differences in DICE similarity of U-net compared to manual segmentation from the second user.

Keywords: Modeling - Anatomical, physiological and pathological, Lung, Computed tomography (CT)
Abstract: Lung biomechanical properties can be used to detect disease, assess abnormal lung function, and track disease progression.In this work, we used computed tomography (CT) imaging to measure three biomechanical properties in the lungs of subjects with varying degrees of chronic obstructive pulmonary disease (COPD): the Jacobian determinant (J), a measure of volumetric expansion or contraction; the anisotropic deformation index (ADI), a measure of the magnitude of anisotropic deformation; and the the slab-rod index (SRI), a measure of the nature of anisotropy (i.e., whether the volume is deformed to a rod-like or slab-like shape). We analyzed CT data from247 subjects collected as part of the Subpopulations and Inter-mediate Outcome Measures in COPD Study (SPIROMICS). The results show that the mean J and mean ADI decrease as disease severity increases, indicating less volumetric expansion and more isotroic expansion with increased disease. No differences in average SRI index were observed across the different levels of disease. The methods and analysis described in this study may provide new insights into our understanding of the biomechanical behavior of the lung and the changesthat occur with COPD.

Keywords: EEG & MEG, Machine learning
Abstract: Brain-computer interfaces (BCIs) based on motor imagery (MI) have played an important role and obtained remarkable achievement in exercise rehabilitation. However, most of the previous researches focused on the upper limb, while many disabled patients need the same technology to assist their rehabilitation training. It is more difficult to detect lower limb MI than upper limb because of the deeper and smaller corresponding sensorimotor cortex. To solve this problem, a new paradigm is proposed to perform waking imagery (WI) for subjects in a virtual environment (VE) to further enhance their brain activities. Furthermore, to decode WI efficiently when facing the low reliable and limited data, we propose a stacked denoising auto-encoder (SDAE) network which is trained on multi-view data obtained in VE. First, the spatial and time-frequency-based features are extracted and fused from the raw data. Second, we use SDAE network to extract the hidden features from the above features. Third, we fuse the previous features and hidden features to train the Softmax classifier. Experimental results on our self-collected data demonstrate that, SDAE network outperforms other deep learning methods in classifying WI in VE.

Keywords: Computer-aided detection and diagnosis (CAD), Brain, Magnetic resonance imaging (MRI)
Abstract: Alzheimer's disease (AD) is a neurodegenerative disorder with progressive impairment of memory and cognitive functions. Sparse coding (SC) has been demonstrated to be an efficient and effective method for AD diagnosis and prognosis. However, previous SC methods usually focus on the baseline data while ignoring the consistent longitudinal features with strong sparsity pattern along the disease progression. Additionally, SC methods extract sparse features from image patches separately rather than learn with the dictionary atoms across the entire subject. To address these two concerns and comprehensively capture temporal-subject sparse features towards earlier and better discriminability of AD, we propose a novel supervised SC network termed Temporally Adaptive-Dynamic Sparse Network (TADsNet) to uncover the sequential correlation and native subject-level codes from the longitudinal brain images. Our work adaptively updates the sparse codes to impose the temporal regularized correlation and dynamically mine the dictionary atoms to make use of entire subject-level features. Experimental results on ADNI-I cohort validate the superiority of our approach.

Keywords: Image segmentation, Brain, Magnetic resonance imaging (MRI)
Abstract: Autoencoder-based approaches for Unsupervised Anomaly Detection (UAD) in brain MRI have recently gained a lot of attention and have shown promising performance. However, brain MR images are particularly complex and require large model capacity for learning a proper reconstruction, which existing methods encounter by restricting themselves to downsampled data or anatomical subregions. In this work, we show that models with limited capacity can be trained and used for UAD in full brain MR images at their native resolution by introducing skip-connections, a concept which has already proven beneficial for biomedical image segmentation and image-to-image translation, and a dropout-based mechanism to prevent the model from learning an identity mapping. In an ablative study on two different pathologies we show considerable improvements over State-of-the-Art Autoencoder-based UAD models. The stochastic nature of the model also allows to investigate epistemic uncertainty in our so-called Skip-Autoencoder, which is briefly portrayed.

Keywords: Machine learning, Image segmentation, Magnetic resonance imaging (MRI)
Abstract: Supervised training of deep neural networks in medical imaging applications relies heavily on expert-provided annotations. These annotations, however, are often imperfect, as voxel-by-voxel labeling of structures on 3D images is difficult and laborious. In this paper, we focus on one common type of label imperfection, namely, false negatives. Focusing on brain lesion detection, we propose a method to train a convolutional neural network (CNN) to segment lesions while simultaneously improving the quality of the training labels by identifying false negatives and adding them to the training labels. To identify lesions missed by annotators in the training data, our method makes use of the 1) CNN predictions, 2) prediction uncertainty estimated during training, and 3) prior knowledge about lesion size and features. On a dataset of 165 scans of children with tuberous sclerosis complex from five centers, our method achieved better lesion detection and segmentation accuracy than the baseline CNN trained on the noisy labels, and than several alternative techniques.

Keywords: Magnetic resonance imaging (MRI), Brain, Machine learning
Abstract: Predicting individual chronological age based on neuroimaging data is very promising and important for understanding the trajectory of normal brain development. In this work, we proposed a new model to predict brain age ranging from 12 to 30 years old, based on structural magnetic resonance imaging and a deep learning approach with reduced model complexity and computational cost. We found that this model can predict brain age accurately not only in the training set (N = 1721, mean absolute error is 1.89 in 10-fold cross validation) but in an independent validation set (N = 226, mean absolute error is 1.96), substantially outperforming the previous published models. Given the considerable accuracy and generalizability, it is promising to further deploy our model in the clinic and help to investigate the pathophysiology of neurodevelopmental disorders.

Keywords: Machine learning, Data Mining, Brain
Abstract: Imaging genetics is a methodology for discovering associations between imaging and genetic variables. Many studies adopted sparse models such as sparse canonical correlation analysis (SCCA) for imaging genetics. These methods are limited to modeling the linear imaging genetics relationship and cannot capture the non-linear high-level relationship between the explored variables. Deep learning approaches are underexplored in imaging genetics, compared to their great successes in many other biomedical domains such as image segmentation and disease classification. In this work, we proposed a deep learning model to select genetic features that can explain the imaging features well. Our empirical study on simulated and real datasets demonstrated that our method outperformed the widely used SCCA method and was able to select important genetic features in a robust fashion. These promising results indicate our deep learning model has the potential to reveal new biomarkers to improve mechanistic understanding of the studied brain disorders.

Keywords: Multi-modality fusion, Graphical models & methods, Machine learning
Abstract: Learning from the multimodal brain imaging data attracts a large amount of attention in medical image analysis due to the proliferation of multimodal data collection. It is widely accepted that multimodal data can provide complementary information than mining from a single modality. However, unifying the image-based knowledge from the multimodal data is very challenging due to different image signals, resolution, data structure, etc.. In this study, we design a supervised deep model to jointly analyze brain morphometry and functional connectivity on the cortical surface and we name it deep multimodal brain network learning (DMBNL). Two graph-based kernels, i.e., geometry-aware surface kernel (GSK) and topology-aware network kernel (TNK), are proposed for processing the cortical surface morphometry and brain functional network. The vertex features on the cortical surface from GSK is pooled and feed into TNK as its initial regional features. In the end, the graph-level feature is computed for each individual and thus can be applied for classification tasks. We test our model on a large autism imaging dataset. The experimental results prove the effectiveness of our model.

Keywords: Blind source separation & Dictionary learning, Magnetic resonance imaging (MRI), Brain
Abstract: We consider adaptive weighted minimax-concave (WMC) penalty as a generalization of the minimax-concave penalty (MCP) and vector MCP (VMCP). We develop a computationally efficient algorithm for sparse recovery considering the WMC penalty. Our algorithm in turn employs the fast iterative soft-thresholding algorithm (FISTA) but with the key difference that the threshold is adapted from one iteration to the next. The new sparse recovery algorithm when used for dictionary learning has a better representation capability as demonstrated by an application to magnetic resonance image denoising. The denoising performance turns out to be superior to the state-of-the-art techniques considering the standard performance metrics namely peak signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM).

Keywords: Pattern recognition and classification, Computer-aided detection and diagnosis (CAD), Nerves
Abstract: Depression affects more than 300 million people around the world and is the leading cause of disability in USA for individuals ages from 15 to 44. The damage of it compares to most common diseases like cancer, diabetes, or heart disease according to the WHO report. However, people with depression symptoms sometimes do not receive proper treatment due to access barriers. In this paper, we propose a method that automatically detects depression using only landmarks of facial expressions, which are easy to collect with less privacy exposure. We deal with the coarse-grained labels i.e. one final label for the long-time series video clips, which is the common cases in applications, through the integration of feature manipulation and multiple instance learning. The effectiveness of our method is compared to other visual based methods, and our method even outperforms multi-modal methods that use multiple modalities.

Keywords: Fetus, Machine learning, Ultrasound
Abstract: The characterization of the fetal cardiac cycle is an important determination of fetal health and stress. The anomalous appearance of different anatomical structures during different phases of the heart cycle is a key indicator of fetal congenital hearth disease. However, locating the fetal heart using ultrasound is challenging, as the heart is small and indistinct. In this paper, we present a viewpoint agnostic solution that automatically characterizes the cardiac cycle in clinical ultrasound scans of the fetal heart. When estimating the state of the cardiac cycle, our model achieves a mean-squared error of 0.177 between the ground truth cardiac cycle and our prediction. We also show that our network is able to localize the heart, despite the lack of labels indicating the location of the heart in the training process.

Keywords: Magnetic resonance imaging (MRI), Heart, Image enhancement/restoration(noise and artifact reduction)
Abstract: Noise and aliasing artifacts arise in various accelerated cardiac magnetic resonance (CMR) imaging applications. In accelerated myocardial T1-mapping, the traditional three-parameter based nonlinear regression may not provide accurate estimates due to sensitivity to noise. A deep neural network-based framework is proposed to address this issue. The DeepT1 framework consists of recurrent and U-net convolution networks to produce a single output map from the noisy and incomplete measurements. The results show that DeepT1 provides noise-robust estimates compared to the traditional pixel-wise three parameter fitting.

Keywords: Ultrasound, Image segmentation, Heart
Abstract: Image segmentation is one of the most important building blocks in many medical imaging applications. Often, it is the first step in any artificial intelligence (AI) assisted diagnosis system. Most convolutional neural network image-to-image segmentation algorithms compute binary mask segmentation and extract the object boundary as the edge of the binary mask always leading to a closed boundary. In this paper, we present a novel image-to-image segmentation algorithm that learns open boundaries. The object delineation is directly learnt by training a U-Net like network on distance map representation of the boundary without any constraints on its shape or topology. We validate the proposed approach on the segmentation of the left atrium in intra-cardiac echocardiography images. For this application, it is important to produce segmentation only where a strong evidence of the anatomy exists. To our knowledge, this is the first work to train a U-Net on distance map ground truth representation for open boundary segmentation.

Keywords: Image segmentation, Heart, Magnetic resonance imaging (MRI)
Abstract: Automated medical image segmentation is an important step in many medical procedures. Recently, deep learning networks have been widely used for various medical image segmentation tasks, with U-Net and generative adversarial nets (GANs) being some of the commonly used ones. Foreground-background class imbalance is a common occurrence in medical images, and U-Net has difficulty in handling class imbalance because of its cross entropy (CE) objective function. Similarly, GAN also suffers from class imbalance because the discriminator looks at the entire image to classify it as real or fake. Since the discriminator is essentially a deep learning classifier, it is incapable of correctly identifying minor changes in small structures. To address these issues, we propose a novel context based CE loss function for U-Net, and a novel architecture Seg-GLGAN. The context based CE is a linear combination of CE obtained over the entire image and its region of interest (ROI). In Seg-GLGAN, we introduce a novel context discriminator to which the entire image and its ROI are fed as input, thus enforcing local context. We conduct extensive experiments using two challenging unbalanced datasets: PROMISE12 and ACDC. We observe that segmentation results obtained from our methods give better segmentation metrics as compared to various baseline methods.

Keywords: Heart, Image reconstruction - analytical & iterative methods, Magnetic resonance imaging (MRI)
Abstract: Perfusion cardiac MRI (CMR) is a radiation-free and non-invasive imaging tool which has gained increasing interest for the diagnosis of coronary artery disease. However, resolution and coverage are limited in perfusion CMR due to the necessity of single snap-shot imaging during the first-pass of a contrast agent. Simultaneous multi-slice (SMS) imaging has the potential for high acceleration rates with minimal signal-to-noise ratio (SNR) loss. However, its utility in CMR has been limited to moderate acceleration factors due to residual leakage artifacts from the extra-cardiac tissue such as the chest and the back. Outer volume suppression (OVS) with leakage-blocking reconstruction has been used to enable higher acceleration rates in perfusion CMR, but suffers from higher noise amplification. In this study, we sought to augment OVS-SMS/MB imaging with a regularized leakage-blocking reconstruction algorithm to improve image quality. Results from highly-accelerated perfusion CMR show that the method improves upon SMS-SPIRiT in terms of leakage reduction and split slice (ss)-GRAPPA in terms of noise mitigation.