**Motivation**: Humans possess a variety of sensory
systems with which we interact with the outside world. When a human
observer views a scene, his or her visual system essentially segments
the scene. This process is very efficient, in the sense that a person
does not have to see a complex scene, but rather a collection of
semantically defined objects. This was the first insight that made me
aware of the possibility of endowing computational methods with the
capability to achieve such segmentation automatically. Although this
research question is well established, a robust, accurate, and
high-performance solution remains a great challenge today. My interest
in this problem space has grown to encompass more than simple
segmentation; I am keenly interested in the study of how semantic
information can be inferred from acquired signals. This was the main
motivation behind my master’s research: automated visual analysis of
sperm motion to perform numerous tedious computations that would be
impractical for a human to realize efficiently.

- Pulmonary nodule detection and segmentation for early diagnosis of lung cancer
- Colon segmentation for virtual colonoscopy and early screening of colorectal cancer
- Segmentation with shape priors

**Joint work with:** Aly
Farag

The segmentation of biomedical images typically deals with partitioning an image into multiple regions representing anatomical objects of interest. A variety of medical image segmentation problems present significant technical challenges, including heterogeneous pixel intensities, noisy/ill-defined boundaries, and irregular shapes with high variability. When I joined the Computer Vision and Image Processing (CVIP) Lab at the University of Louisville (UofL) to pursue my Ph.D., I applied my background in image segmentation to the problem of pulmonary nodule detection and classification for the early diagnosis of lung cancer.

Lung cancer in the United States account for 30% of all cancer-related deaths, resulting in over 160,000 deaths per year, which is more than the annual deaths for colon, breast, prostate, ovarian and pancreatic cancers combined. The survival of lung cancer is strongly dependent of diagnosis.

At the CVIP Lab, we developed a Computer-Assisted Diagnosis (CAD) system that consisted of four main stages. The preprocessing stage consisted of removing any noise artifacts that may have developed onto the CT images during scanning with minimal information loss. The goal of the image segmentation step was to achieve accurate segmentation while maintaining the details of the lung anatomy from low-dose chest CT scans. In particular, it was essential that the pulmonary nodules inside the lungs as well as on the boundary regions be maintained for subsequent analyses.

*A block diagram of the major steps involved in computer-based
analysis of LDCT of the chest in order to detect and classify doubtful
lung nodules.*

*Average histogram of a CT slice from the ELCAP dataset. The
average histogram of CT slices of the ELCAP database. Lung region
(parenchyma) and fat/muscle region constitute two dominant peaks in the
histogram, in order to separate the lung region a threshold is chosen to
maximize the separation between these two peaks.*

*Schematic diagram of the lung segmentation algorithm.*

Nodule detection step was where candidate nodules were identified using object modeling and recognition techniques. The approach of nodule detection hinged mainly on proper modeling of nodule templates and much less on the computational approach to carry out the detection.

*The distribution of the radial distance from the centroid of the
nodules. The bars are one standard deviation off the mean values of each
nodule type. Note the exponential behavior of the radial distance, and
that it diminishes after a distance of 10.*

*The most common parametric nodule models in 2D are circular and
semi-circular. In 3D the corresponding models would be spherical and
hemi-sphere or cups. This shows a few examples of such templates. For
the 2D case the circular and semi-circular parametric templates
(isotropic and non-isotropic) are used. The isotropic templates are
defined in terms of the radius (size) and the gray level distribution as
a circular symmetric Gaussian function while non-isotropic templates are
defined by radius, gray level distribution and orientation.*

We have also developed a data-driven approach o obtain representations that depict the realistic shape and texture properties of the various nodule types found in LDCT scans. The four main types of nodules we have been concerned with were: (1) Well-circumscribed where the nodule is located centrally in the lung without being connected to vasculature; (2) Vascularized where the nodule has significant connection(s) to the neighboring vessels while located centrally in the lung; (3) Juxta-pleural where a significant portion of the nodule is connected to the pleural surface; and (4) Pleural-tail where the nodule is near the pleural surface, connected by a thin structure.

*Three cropped nodules from the four nodule types.
Well-circumscribed (1st row) vascular (2nd row), juxta-pleural (3rd row)
and pleural-tail (4th row) nodule types.*

The mean nodule templates for these four nodule types generated using an Active Appearance Modeling (AAM) approach is depicted below. These models were used in the detection process which has shown much improvement over other methods found in the literature.

*Generation of data-driven nodule models.*

The final step, nodule classification took the candidate nodules located and identified or categorized these candidates by types, location, morphology, etc. but ultimately and the most important identification is benign and/or malignant.

**Related publications:**

Amal Farag, **Shireen Y. Elhabian**, James Graham, Aly
Farag, Salwa Elshazly, Robert Falk, Hani Mahdi, Hossam Abdelmunim, Sahar
Al-Ghaafary. Modeling of
the Lung Nodules for Detection in LDCT Scans. *Proc. of the 32nd
IEEE Engineering in Medicine and Biology Society (EMBC)*, 2010,
pp. 3618-3621.

Amal Farag, **Shireen Y. Elhabian**, Salwa Elshazly, Aly
Farag. Quantification
of nodule detection in chest CT: A clinical investigation based on the
ELCAP study. *In Proceeding of Second International Workshop on
Pulmonary Image Processing in conjunction with MICCAI*, pp. 149-160,
2009.

**Shireen Y. Elhabian**, Amal Farag, Salwa Elshazly, Aly
Farag. Sensitivity
of Template Matching for Pulmonary Nodule Detection: A Case Study.
*In IEEE Biomedical Engineering Conference (CIBEC)*. Cairo
International, pp. 1-4, 2008.

**Shireen Y. Elhabian**, Hossam Abd EL Munim, Salwa
Elshazly, Aly A Farag, Mohamed Aboelghar. Experiments on
Sensitivity of Template Matching for Lung Nodule Detection in Low Dose
CT Scans. *IEEE International Symposium on Signal Processing and
Information Technology (ISSPIT)* 2007, 1040-1046.

**Joint work with:** Aly
Farag

Colorectal cancer, which includes cancer of the colon, rectum, anus, and appendix, is the third most common form of cancer and the second leading cause of death among cancers in the western world. Most colorectal cancers begin as a polyp: a small, superficial growth arising from the wall of the colon. As a polyp grows, it can develop into a cancer that invades and spreads. At the Computer Vision and Image Processing (CVIP) Lab at the University of Louisville (UofL), we developed a fully automated framework for accurate 3D colon tissue segmentation in CT colonography using adaptive level sets method with a focus on early screening of colorectal cancer. Subsequent steps involved building the 3D colon model by searching the iso-surface, and finally extract the 3D centerline for colon 3D navigation.

*A sample CT colon slice showing typical challenges confronting colon
segmentation.*

For the segmentation step, we generalized the global/convex continuous minimization problem of the active contour model to the 3D case using Mumford and Shah’s functional model and Chan and Vese’s model of active contours without edges. Furthermore, we incorporated anatomical features with the 3D region growing in a postprocessing stage to discard non-colon parts such as spine and bowels while maintaining all colon segments. The proposed framework had an average accuracy level of approximately 99%. We demonstrated that this approach outperformed conventional approaches, including graph cuts (discrete optimization) and adaptive level sets (non-convex continuous optimization), with regard to accuracy, sensitivity, specificity, and speed of convergence.

*Schematic diagram and sample result of the proposed 3D colon
segmentation framework.*

*Sample result of a poorly distended colon. A 2D slice is shown
with close-up views on challenging areas such as haustral folds.
Quantitative results are shown for adaptive level sets and graph cuts
along with our results according to Ac(accuracy), Sn(sensitivity),
Sp(specificity), t(time) and iter(number of iterations).*

**Related publications:**

Marwa Ismail, **Shireen Y. Elhabian**, Aly Farag and
Gerald W. Dryden. Fully Automated 3D
Colon Segmentation for Early Detection of Colorectal Cancer Based on
Convex Formulation of the Active Contour Model. *IEEE Computer
Society Conference on Computer Vision and Pattern Recognition Workshops
(CVPRW)*, pp. 58-63. IEEE, 2012.

Marwa Ismail, **Shireen Y. Elhabian**, Aly Farag, Gerald
Dryden, Albert Seow. 3D Automated Colon
Segmentation for Efficient Polyp Detection. *In Biomedical
Engineering Conference (CIBEC)*, 2012 Cairo International,
pp. 48-51. IEEE, 2012.

**Joint work with:** Ross Whitaker, Joshua Cates, and Nassir
F. Marrouche

My current research in this area with Ross Whitaker at the University of Utah focuses on optimal segmentations within a statistical framework that combines image data with priors on anatomical structures. We began working with local shape priors including smoothness and spatial coherency. Recently, we have been working on the inclusion of global shape priors within a Bayesian estimation framework for image segmentation in which generative models of shapes play an important role to effectively learn priors that guide the image segmentation process. We demonstrated our approach on the left atrial wall segmentation from late-gadolinium enhancement (LGE) MRI, which has been shown to be effective for identifying myocardial fibrosis in the diagnosis of atrial fibrillation.

*Left artrium wall segmentation challenges: Slices of LGE-MRI
images showing the challenges in segmentation. The left two 2D slices
are shown without manual delineations of the LA wall, and the right two
slices show manual delineations.*

We developed a surface-based image segmentation scheme, termed as
*ShapeCut*, a shape-based generative model for extracting
multiple surfaces from a given image. ShapeCut is modeled by
incorporating global shape information within the Bayesian framework,
thus biasing the solution toward the desired shape. However, to
accommodate a subclass of shapes that are not captured by means of the
global shape priors, due to the smaller sample size, the proposed method
introduces local shape priors in the form of MRFs. The optimization of
the derived model is done in two phases that are performed in an
iterative fashion: one for a multi-column graph-based multisurface
update and the other for closed form-based global shape refinement. The
results demonstrated the effectiveness of our approach in the presence
of weak boundaries, contrast variation and a low signal-to-noise
ratio.

*Schematic diagram of ShapeCut: the proposed graph-based object
segmentation with shape priors.*

ShapeCut differs from many other Bayesian methods in that the underlying search space is an approximation of a continuous parameterization of the set of possible surfaces. We discretize the underlying continuous parameterization in the form of a special form of geometric graph structure, in order to simultaneously estimate multiple, interacting surfaces in a globally optimal manner. The below figure depicts the strategy representing the continuous parameterization in the form of a discrete graph whose design involves a set of nested layers and columns. Here, each layer maintains a similar topological structure to that of the desired surface, and each column necessarily ensures the estimated surface to pass though it. For the 2D surface estimation, these layers represent 2D discrete contours, and in the case of 3D, these layers become meshes.

*(a) Continuous parameterization of the surface estimate \(\mathcal{S}\), (b) discrete approximation
of the underlying continuous parameterization within which the surface
estimation takes place and (c) overlay of discrete grid on a given image
and intensity profiles, \(p_{i,j}\)
(yellow) at different grid points \(x_{i,j}\).*

The shape prior in ShapeCut is a generative model for surfaces. For this work we use a linear model and an associated set of latent variables, \(\beta\), to represent global shape information. This linear model is learned over training shapes, and captures low-dimensional properties of shape representation. The figure below shows a simplified geometric interpretation of the shape model. The linear subspace, which is learned from training data, is relatively low-dimensional, but allows the segmentation algorithm to operate effectively when parts of the anatomy are not well delineated in the image data, typically because of low-contrast and high-correlated and/or uncorrelated noise. A shape is generated from a position \(\beta\) on this low-dimensional shape representation (i.e., shape parameters). To account for the actual complexity of the left atrium, the generative shape model allows smooth deviations or offsets from the low-dimensional shape distribution, formulated as a Markov random field (MRF) prior on their configuration.

*(a) Training shapes implicitly represented by distance
transforms, (b) projection of shapes onto a linear subspace and (c)
estimated surface \(\mathcal{S}\)
derived from the deviation, \(\mathcal{S}_o\), of the base shape \(\mathcal{S}_\beta\).*

ShapeCut is an effective method for the automatic extraction of multiple coupled surfaces such as the left atrium wall. Compared to the manual segmentations that typically take more than an hour and requires a human expert, ShapeCut achieves significant time improvement (average run-time of 2.3 minutes for the left atrium wall extraction) apart from providing similar results, except around vein regions, not only qualitatively and quantitatively but also from a clinical perspective.

*Anterior-posterior/posterior-anterior views of LGE-MRIs depicting
fibrosis patterns in manual (top) vs the proposed method (bottom) wall
regions. Fibrosis regions are displayed in green and healthy wall
regions in blue.*

Recently, we formulated the shape prior as a mixture of Gaussians and
learn the corresponding parameters in a high-dimensional shape space
rather than pre-projecting onto a low-dimensional subspace. We used deep
autoencoders to capture the complex intensity distribution while
avoiding the careful selection of hand-crafted features. For
segmentation, we treated the identity of the mixture component as a
latent variable and marginalize it within a generalized
expectation-maximization framework. We presented a conditional
maximization-based scheme that alternates between a closed-form solution
for component-specific shape parameters that provides a global
update-based optimization strategy, and an intensity-based energy
minimization that translates the global notion of a nonlinear shape
prior into a set of local penalties.

*(a) Low-dimensional subspace representation of high-demensional
shape data. Each sample in the training set is represented by a single
point with color representing different components of the GMM. An
example segmentation from each component is included to illustrate
differences. (b) Intensity gradients across the segmentation surface are
taken at each point location.*

*Example results for 2 different samples with point by point error
mapped onto segmentation surface. AE = autoencoder, GMM = mixture
modeling, Original = neither (original ASM method).*

**Related publications:**

Gopalkrishna Veni, **Shireen Y. Elhabian**, and Ross T.
Whitaker. ShapeCut:
Bayesian Surface Estimation Using Shape-Driven Graph. *Medical
Image Analysis* (2017).

Gopalkrishna Veni, **Shireen Y. Elhabian**, Ross
Whitaker. A
Bayesian Formulation of Graph-Cut Surface Estimation With Global Shape
Priors. *In IEEE 12th International Symposium on Biomedical
Imaging (ISBI)*, 2015, pp. 368-371.

Tim Sodergren, Riddhish Bhalodia, Ross Whitaker, Joshua Cates, Nassir
Marrouche, and **Shireen Y. Elhabian**. Mixture Modeling of
Global Shape Priors and Autoencoding Local Intensity Priors for Left
Atrium Segmentation. *STACOM-MICCAI: Statistical Atlases and
Computational Modeling of the Heart workshop*, 2018.

Copyright © 2022 Shireen Y. Elhabian. All rights reserved.