Image Analysis

Neighborhood Looking Glass: 360 Degree Automated Characterization of the Built Environment for Neighborhood Effects Research

Tolga Tasdizen
This proposal represents a vertical advancement in neighborhood effects research, producing for the first time, national neighborhood indicators of the built environment. Thus far, only local studies have been conducted due to the resource-intensive nature of site visits to conduct assessments of community features and also manual annotations of street images. With the recent advancement of computer vision and the emergence of massive sources of image data, we will leverage our team’s abilities to develop a data collection strategy utilizing geographic information systems to assemble a national collection of Google Street View images of all road intersections and street segments in the United States. We will utilize this data bank, and develop informatics algorithms to produce neighborhood summaries of built environment that have been theoretically and empirically identified to be important for health outcomes. After the creation of Neighborhood Looking Glass, we will conduct investigations into the impact of neighborhood environments on health utilizing medical records from hundreds of thousands of patients and accounting for predisposing characteristics in analyses. Our investigative team (comprised of experts in the field of epidemiology, computer vision, bioinformatics, and computer science) is uniquely suited to implement the study aims.
Our Specific Aims are: 1) Develop informatics techniques to produce neighborhood quality indicators; 2) Measure the accuracy of data algorithms and construct an interactive geoportal for neighborhood data visualization and data sharing, 3) Utilize Neighborhood Looking Glass and a large collection of medical records from Intermountain Healthcare to investigate neighborhood influences on the risk of obesity and substance abuse. The epidemic rise in chronic health conditions is recent and as such suggests its cause is social, cultural, and constructed rather than purely biological. Thus, we have the possibility of intervening on the environment to better support health. Recent studies suggest that the current cohort of young adults may face historically high cardiovascular disease risk and chronic disease burden. Our substantive investigation of the impact of neighborhood factors on chronic conditions will contribute further to the understanding of contextual influences on the health of this cohort at the forefront of a chronic disease epidemic. Moreover, the dramatic rise in overdoses, accidental poisonings, and mental health issues contributing to premature mortality warrants further investigation into risk-inducing environmental factors for substance abuse. Neighborhood Looking Glass will be a significant benefit to neighborhood effects researchers, harnessing the largely untapped potential of street image data to capture built environment characteristics. Results can be utilized to inform population-based strategies to reduce health disparities and improve health.

The epidemic rise in obesity, related chronic diseases, and substance abuse in recent decades signal the importance of structural forces and social processes, but the dearth of data on contextual factors limits the investigation of multilevel effects on health. The development of the Neighborhood Looking Glass will be a significant benefit to neighborhood effects researchers, harnessing the largely untapped potential of street image data to capture built environment characteristics with potential impact on health. Results from our project can be utilized to inform system-wide and local strategies to improve community health.

Shapeworksstudio: An Integrative, User-Friendly, and Scalable Suite for Shape Representation and Analysis

Shireen Youssef Elhabian
The morphology (or shape) of anatomical structures forms the common language among clinicians, where abnormalities in anatomical shapes are often tied to deleterious function. While these observations are often qualitative, finding subtle, quantitative shape effects requires the application of mathematics, statistics, and computing to parse the anatomy into a numerical representation that will facilitate testing of biologically relevant hypotheses. Particle-based shape modeling (PSM) and its associated suite of software tools, ShapeWorks, enable learning population-level shape representation via automatic dense placement of homologous landmarks on image segmentations of general anatomy with arbitrary topology. The utility of ShapeWorks has been demonstrated in a range of biomedical applications. Despite its obvious utility for the research enterprise and highly permissive open-source license, ShapeWorks does not have a viable commercialization path due to the inherent trade-off between development and maintenance costs, and a specialized scientific and clinical market. ShapeWorks has the potential to transform the way researchers approach studies of anatomical forms, but its widespread applicability to medicine and biology is hindered by several barriers that most existing shape modeling packages face. The most important roadblocks are (1) the complexity and steep learning curve of existing shape modeling pipelines and their increased computational and computer memory requirements; (2) the considerable expertise, time, and effort required to segment anatomies of interest for statistical analyses; and (3) the lack of interoperable implementations that can be readily incorporated into biomedical research laboratories. In this project, we propose ShapeWorksStudio, a software suite that leverages ShapeWorks for the automated population-/patient-level modeling of anatomical shapes, and Seg3D--a widely used open-source tool to visualize and process volumetric images--for flexible manual/semiautomatic segmentation and interactive manual correction of segmented anatomy.

In Aim 1, we will integrate ShapeWorks and Seg3D in a framework that supports big data cohorts to enable users to transparently proceed from image data to shape models in a straightforward manner.

In Aim 2, we will endow Seg3D with a machine learning approach that provides automated segmentations within a statistical framework that combines image data with population-specific shape priors provided by ShapeWorks.

In Aim 3, we will support interoperability with existing open-source software packages and toolkits, and provide bindings to commonly used programming languages in the biomedical research community. To promote reproducibility, we will develop and disseminate standard workflows and domain-specific test cases. This project combines an interdisciplinary research and development team with decades of experience in statistical analysis and image understanding, and application scientists to confirm that the proposed developments have a real impact on the biomedical and clinical research communities. Our long-term goal is to make ShapeWorks a standard tool for shape analyses in medicine, and the work proposed herein will establish the groundwork for achieving this goal.

Public Health Relevance
ShapeWorks is a free, open-source software tool that uses a flexible method for automated construction of statistical landmark-based shape models of ensembles of anatomical shapes. ShapeWorks has been effective in a range of applications, including psychology, biological phenotyping, cardiology, and orthopedics. If funded, this application will ensure the viability of ShapeWorks in the face of the ever-increasing complexity of shape datasets and support its availability to biomedical researchers in the future, as well as provide opportunities for use in a wide spectrum of new biological and clinical applications, including anatomy reconstruction from sparse/low-dimensional imaging data, large-scale clinical trials, surgical planning, optimal designs of medical implants, and reconstructive surgery.

Data-Driven Shape Analysis for Quantitative Severity Stratification in Patients with Metopic Craniosynostosis

Ross Whitaker
Craniosynostosis affects close to one in 2000 newborns and causes growth restriction perpendicular to the affected suture. Metopic craniosynostosis is the second most common form of craniosynostosis. The metopic suture is an important sight of cranial growth as the brain rapidly expands in the first year of life. Patients affected by metopic craniosynostosis will present in the first few months of life with varying degrees of narrowing of the forehead and brow, a triangular shaped head, and an abnormal eye position. Surgery is recommended early in childhood to normalize the head shape and expand the restricted skull to prevent complications such as headaches, cognitive impairment, and visual disturbances including blindness. Imaging with computed tomography (CT) is employed to confirm new diagnoses of metopic craniosynostosis and, together with the physical exam, is used in a descriptive and qualitative manner to assess the degree of head shape abnormality. Several methods have been employed to interpret the information provided in the CT scans to allow surgeons to utilize data for surgical decision making. However, these indices reduce the complex three-dimensional skull dysmorphology into isolated measurements of angles or proportions, require detailed calculations to perform, and no universally accepted standard has emerged so far despite significant research efforts and clinical motivation. In this grant proposal, we aim to increase our understanding of the cranial dysmorphology in patients with metopic craniosynostosis by employing latest results from statistical shape modeling and deep learning. Specifically, we will build a statistical shape model of pediatric skulls from CT images of patients with metopic craniosynostosis as well as a group of normal controls capturing normal phenotypical shape variations. The distance of a new shape from the normative shape space will represent the proposed Shape Normality Metric (SNM). The SNM will be validated against ratings from experts in the surgical community (current standard of care) who will be asked to assess the dysmorphology of the skulls in our database. To avoid surgeons' subjective bias, we will aggregate their response using statistical methods that compensate for potential individual bias. Finally, to streamline data collection for future research we will develop a head-shape portal that will allow users to upload CT scans of their patients and the system will automatically calculate the SNM. By developing a severity metric that encompasses the entire extent of dysmorphology in metopic craniosynostosis and establishing a head-shape portal, we will improve our understanding of the spectrum of metopic craniosynostosis, aid in pre-operative and surgical decision making, enable future research, and help facilitate longitudinal outcomes assessments and multi-center communication and collaboration.

Public Health Relevance
This grant proposal aims to improve our understanding of the head shape anomaly associated with metopic craniosynostosis by using recent results from statistical shape analysis and deep learning, with the goal of developing an objective metopic cranioynostosis severity scale. Different from previously proposed metrics, our approach evaluates the entire shape as a whole. With this information, surgeons will be able to objectively determine how severely affected their patients are and will be better able to tailor their interventions to the needs of their individual patients. Additionally, surgeons will be able to better communicate with each other and study the effects of surgical intervention on their patients which will improve patient care in the long run.

A Scalable Non-Intrusive Image Annotation Method Using Eye Tracking for Training Deep Learning Models in Radiology

Tolga Tasdizen
Machine learning (ML) and artificial intelligence have recently emerged as powerful techniques that can augment radiology interpretations and show promise for improving patient outcomes. One of the ways for ML to make a significant impact on health care is in improving the evaluation of high-volume, low-cost exams for early signs of a wide variety of diseases. The routine chest x-ray is an opportunity for screening for diseases, including cancer, chronic obstructive pulmonary disease (COPD), pneumonia and congestive heart failure. For instance, lung cancer is the most common cause of cancer death in the US, and is typically diagnosed at a higher stage than most other cancers leading to low survival rates. The National Lung Screening Trial reported that low dose computed tomography (LDCT) screening resulted in a 20% reduction in lung cancer mortality; however, few eligible people actually undergo LDCT screening. Meanwhile, chest x-rays continue to be the most common form of imaging worldwide. Improved detection from x-rays can direct patients to LDCT. COPD is another important disease that is often under-diagnosed. People with COPD are at increased risk of lung cancer and respiratory infections, or exacerbations, which are associated with higher morbidity and mortality. Furthermore, a chest x-ray may show poorly-defined regions of consolidation that are concerning for pneumonia. Medical attention is required to treat an infection or evaluate for other cause. More generally, methods to detect disease on chest x-rays can be extended to cardiomegaly, pulmonary edema and pleural effusions which are seen in congestive heart failure. Improved detection can direct patients to medical care. Convolutional neural networks (CNN), a highly successful ML model, can be applied to chest x-ray images. However, few annotated medical datasets exist that are sufficiently large to train CNNs. Furthermore, it has been shown that bounding boxes used to localize disease can be incorporated into the training of CNNs and significantly increase their accuracy. Unfortunately, medical datasets with such localized annotations are even rarer and are very limited in the number of cases due to the time-consuming process of creating bounding boxes by radiologists. We propose an innovative integrated approach using eye tracking, speech recording and novel vision and language models to create localized annotations in a manner that is non-intrusive to the workflow of the radiologist. The novelty of our approach is in the use of eye tracking during routine radiological reading. The challenge is to overcome the relatively ambiguous nature of eye tracking information compared to bounding boxes which provide definitive information about abnormalities. To address this challenge, we will also design new CNN architectures and learning algorithms that can use eye tracking and additional information such as pupil dilation and fixation duration. The proposed methodology can easily scale up to create very large datasets without generating additional workload for radiologists. Furthermore, deployed in the reading room, it could provide a continuous stream of annotated images to expand training sets.

Public Health Relevance
Machine learning and artificial intelligence hold the potential to improve patient outcomes by enhancing the evaluation of low cost, routine chest x-rays for early imaging abnormalities associated with diseases such as lung cancer, chronic obstructive pulmonary disease and pneumonia and making recommendations for follow-up care. To realize their full potential, machine learning algorithms require training on very large datasets with localized annotations for abnormalities. We propose a novel data collection methodology using eye tracking that is scalable, non-intrusive to the workflow of the radiologist, and can generate very large datasets with localized annotations for the development of novel machine learning algorithms to address significant medical problems.

The Space of Riemannian Metrics for the Statistical Analysis of the Human Connectome

Sarang Joshi
The human brain is one of the most complex biological geometrical objects. The Human Connectome Project aims to make available an unparalleled compilation of neural functional and structural imaging data from healthy adults. Data from over 900 subjects has already been released. The principal data provided by the Human Connectome Project are diffusion-weighted MRI and functional MRI. Due to the amount and complexity of the data generated in this project, new techniques to analyze, compare and represent these data are needed, which is the motivation and driving force for the research outlined in this proposal. This collaborative project has three fundamental goals: (1) to further develop the mathematical theory of geometrical statistics, in particular the role of the infinite-dimensional manifold of all Riemannian metrics; (2) to develop practical tools for the statistical study of the connectivity of the human brain; and (3) to demonstrate the utility of the developed techniques for the segmentation and parcellation of the thalamus and other subareas of the subcortical gray matter that are not visible in structural MRI.

This project will develop for the first time statistical techniques on the infinite-dimensional manifold of Riemannian metrics. The project team believes that the space of Riemannian metrics is the natural framework for analyzing the variability of the architecture of the human brain. Diffusion-weighted MRI allows the investigators to model an individual human brain as a Riemannian manifold with axonal connections that are geodesic curves of an appropriate metric. The team will study the space of all Riemannian metrics and develop methods based on geometrical statistics for the analysis of the whole population. An immediate practical application of the techniques developed will be the parcellation of the thalamus based on thalamocortical connectivity. The internal architecture of the thalamus is not visible in standard structural MRI but rather is defined via the connections to the different areas of the cortex. In this project, the investigators will partition the thalamus by projecting the functional partition of the cortex onto the thalamus via the connectomics. The aim is to use geometric statistical mapping methods to produce a statistically informed partition of an individual patient's thalamus. The primary driving motivation is to eventually improve outcomes of deep brain stimulation as a therapy for essential tremor, in which the thalamus is the primary target. The subcortical white matter is also implicated in many neurological disorders, such as ischemic vascular disease, Huntington's, Multiple Sclerosis, and HIV/AIDS dementia. The PIs envision that the statistical techniques developed for qualifying the detailed architecture of the white matter in the normal population will have implications for all these diseases. This project will provide novel analytical tools to unravel the mysteries of the human brain.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Validation and Translation of a Non-Invasive, Mr-Guided Breast Cancer Therapy

Sarang Joshi
The treatment of early stage, localized breast cancer has evolved over several decades from highly invasive techniques to minimally invasive, breast-conserving therapies. Our breast-specific magnetic resonance-guided focused ultrasound system (the Muse MRgFUS System) non-invasively delivers focused energy deep inside the body under high-resolution image guidance. The technical advances we have made poise MRgFUS as a nearly ready non-invasive ablation treatment for localized breast disease. The final steps for its clinical translation include studies to compare, optimize and carefully validate treatment planning, real-time monitoring and assessment techniques against the best available standard metrics, which include invasive temperature probes, hydrophone measurements and histopathology. This proposal pairs experienced academic investigators at the University of Utah with industry investigators at Image Guided Therapy. Combining the academic partner's scientific expertise and technological knowledge with the system integration and regulatory expertise of the industry partner will facilitate translation of this exciting technology. This proposal will develop and validate the remaining elements necessary to translate the Muse System into clinical care including: 1) accurate MRgFUS treatment planning based on a rapid ultrasound beam modeling method, 2) a comprehensive breast-focused volumetric MR thermometry method, and 3) a novel MRI-registered whole mount histology technique that compares MRI metrics to histopathological analysis. We will integrate each of these validated elements in a clinic-ready, treatment software control environment and will evaluate the entire process in a Phase I treat-and-resect clinical trial targeting unifocal invasive breast cancer tumors. The validation and translation of the Muse System in this proposal will provide an exciting new image-guided noninvasive treatment option for breast cancer patients.

Public Health Relevance
The validation of breast-specific treatment planning, monitoring and assessment techniques will accelerate translation of a magnetic resonance-guided focused ultrasound system towards the efficient, non-invasive treatment of localized breast cancer. Incorporating gold standard validated techniques in a clinic-ready planning and monitoring software package will enable a Phase I clinical trial to verify this non-invasive therapy can achieve efficacy within acceptable safety limits.

Anatomy Directly from Imagery: General-Purpose, Scalable, and Open-Source Machine Learning Approaches

Shireen Youssef Elhabian
The form (or shape) and function relationship of anatomical structures is a central theme in biology where abnormal shape changes are closely tied to pathological functions. Morphometrics has been an indispensable quantitative tool in medical and biological sciences to study anatomical forms for more than 100 years. Recently, the increased availability of high-resolution in-vivo images of anatomy has led to the development of a new generation of morphometric approaches, called statistical shape modeling (SSM), that take advantage of modern computational techniques to model anatomical shapes and their variability within populations with unprecedented detail. SSM stands to revolutionize morphometric analysis, but its widespread adoption is hindered by a number of significant challenges, including the complexity of the approaches and their increased computational requirements, relative to traditional morphometrics. Arguably, however, the most important roadblock to more widespread adoption is the lack of user-friendly and scalable software tools for a variety of anatomical surfaces that can be readily incorporated into biomedical research labs. The goal of this proposal is thus to address these challenges in the context of a flexible and general SSM approach termed particle-based shape modeling (PSM), which automatically constructs optimal statistical landmark-based shape models of ensembles of anatomical shapes without relying on any specific surface parameterization. The proposed research will provide an automated, general-purpose, and scalable computational solution for constructing shape models of general anatomy.

In Aim 1, we will build computational and machine learning algorithms to model anatomies with complex surface topologies (e.g., surface openings and shared boundaries) and highly variable anatomical populations.

In Aim 2, we will introduce an end-to-end machine learning approach to extract statistical shape representation directly from images, requiring no parameter tuning, image pre-processing, or user assistance.

In Aim 3, we will provide intuitive graphical user interfaces and visualization tools to incorporate user-defined modeling preferences and promote the visual interpretation of shape models. We will also make use of recent advances in cloud computing to enable researchers with limited computational resources and/or large cohorts to build and execute custom SSM workflows using remote scalable computational resources. Algorithmic developments will be thoroughly evaluated and validated using existing, fully funded, large-scale, and constantly growing databases of CT and MRI images located on-site. Furthermore, we will develop and disseminate standard workflows and domain-specific use cases for complex anatomies to promote reproducibility. Efforts to develop the proposed technology are aligned with the mission of the National Institute of General Medical Sciences (NIGMS), and its third strategic goal: to bridge biology and quantitative science for better global health through supporting the development of and access to computational research tools for biomedical research. Our long-term goal is to increase the clinical utility and widespread adoption of SSM, and the proposed research will establish the groundwork for achieving this goal.

Public Health Relevance
This project will develop general-purpose, scalable, and open-source statistical shape modeling (SSM) tools, which will present unique capabilities for automated anatomy modeling with less user input. The proposed technology will introduce a number of significant improvements to current SSM approaches and tools, including the support for challenging modeling problems, inferring shapes directly from images (and hence bypassing the segmentation step), parallel optimizations for speed, and new user interfaces that will be much easier and scalable than the current tools. The proposed technology will constitute an indispensable resource for the biomedical and clinical communities that will enable new avenues for biomedical research and clinical investigations, provide new ways to answer biologically related questions, allow new types of questions to be asked, and open the door for the integration of SSM with clinical care.

Machine Learning and Signature Analysis of Nuclear Forensic Data

Tolga Tasdizen
The development of uranium oxide physical and chemical signatures is critical to the field of nuclear forensic analysis. Qualitative morphological parameters provide supplementary information in nuclear forensic investigations, but tell a limited portion of an unknown sample’s story. There is a major need for quantitative parameters that can rapidly determine whether differences between an unknown sample and a standard are statistically significant. It would be even more beneficial if these parameters could elucidate not just the starting material speciation, but the processing conditions experienced by the sample. Accounting for storage and temporal effects further accounts for the total process history of an unknown sample. In the future, one could see a quantitative morphological database that expedites the attribution process. We propose to develop a data processing pipeline that streamlines the analysis of complex nuclear forensics data and statistically correlates data across multiple techniques. This pipeline will build upon existing computational tools in the nuclear forensics community, such as the Morphological Analysis for Material Attribution (MAMA) software developed by Los Alamos National Laboratory (LANL), to characterize particle morphologies. The proposed fundamental science in an academic setting coupled with technical guidance from Pacific Northwest National Laboratory (PNNL) will instruct and inform the next generation of nuclear scientists and engineers.

Multi-Tiered Carbon Monitoring System

Sarang Joshi
Reducing CH4 and fossil fuel CO2 emissions remains a top climate mitigation priority for stakeholders around the world. States such as California have committed to ambitious GHG stabilization targets. State agencies such as the California Air Resources Board (CARB) are strongly motivated to verify emissions and inform policy formulation at the scale of major emitting regions, air basins and cities. Additionally, private companies such as Chevron have expressed interest in better facility-scale emissions data to reduce their greenhouse gas footprints and product loss. Meanwhile several foundations such as the Rocky Mountain Institute (RMI) are working to establish trusted climate data initiatives through public-private partnerships. A common feature of these interests is a focus on facility-scale point source emitters and their contribution to local emission budgets to prioritize mitigation efforts. A common challenge is that CH4 and CO2 emissions data at those spatial scales is currently sparse, inaccurate or non-existent. There is an urgent need to provide CH4 and CO2 data and analytics that are trusted, timely and at spatial scales relevant to decision making. A tiered observational strategy and integrated data analysis framework have the potential to leverage emerging and planned airborne and satellite remote sensing capabilities to address these challenges and stakeholder needs (ultimately in key regions globally).

We propose to build on the success of our Prototype Methane Monitoring System for California (CMS-2015-Duren) and Megacities Carbon Project to develop and test a Multi-tiered CH4 (and as a secondary goal: CO2) monitoring system for a broader set of high emitting regions and priority emission sectors in the US. In year 1 of the project we plan to conduct CH4 and CO2 point source surveys of key regions and sectors in California, the Permian basin in Texas and New Mexico, and major oil and gas infrastructure centers along the Gulf Coast with NASA's Next Generation Airborne Visible/Infrared Imaging Spectrometer (AVIRIS-ng) together with coordinated snap-shot mode CO2 observations from the Orbiting Carbon Observatory-3 (OCO-3) and routine CH4 observations from the Sentinel-5 Precursor/TROPOMI satellite. In years 2 and 3 we will generate CH4 (goal: CO2) regional and point source emission estimates in those regions that leverage and extend multi-scale estimation techniques previously prototyped in California. This will allow stakeholders to place facility scale emissions into context with regional emissions. If selected, this project will benefit from additional funding from RMI to support more airborne surveys and data product development. It also benefits from in-kind contributions from other collaborators. Our stakeholders including RMI, CARB and Chevron have indicated an interest in evaluating and potentially adopting the methods, tools and data products developed by this project for infusion into their decision frameworks. Finally, we also plan to leverage existing surface measurements from our own Megacities Carbon Project and collaborators in the southern San Joaquin Valley and potentially the Permian basin, Salt Lake City and the Uintah Basin to help validate emission estimates.

Time-Dependent Analysis of SPT Microscopy Data

Valerio Pascucci