**Motivation**: Various vision tasks are usually
confronted by appearance variations due to changes in illumination. For
instance, in a recognition system, it has been shown that the
variability in the appearance of human face is dominated by changes to
lighting conditions rather than the individual’s identity. My passion
encompassed the extraction of information about the contents of an image
based on an understanding of the underlying physics that governs how the
image was formed. Theoretically, due to the arbitrariness of the
lighting function, the space of all possible images of a fixed-pose
object under all possible illumination conditions has an infinite
dimension. Nonetheless, it has been proven that the set of images of a
convex Lambertian surface under distant illumination lies near a
low-dimensional linear subspace. This result was also extended to
include non-Lambertian objects with non-convex geometry. As such, vision
applications, concerned with the recovery of illumination, reflectance,
and/or surface geometry from images, would benefit from a
low-dimensional generative model that captures appearance variations
with regard to illumination conditions and surface reflectance
properties.

In my Ph.D. research, I proposed an analytic formulation of appearance subspace construction to capture the full behavior of complex illumination and real-world reflectance, and in the process accounted for the hemispherical nature of surface reflectance functions to model physically plausible surface materials. The essence of my work in this area is phenomenological reflectance models that capture appearance variations through mathematical abstraction of the reflection process. In particular, the harmonic expansion of the image irradiance equation can be used to derive an analytic subspace to represent images under fixed-pose, but different illumination conditions.

*Image formation triology, light source(s) are used to illuminate
a 3D object in space which will be captured by an imaging system,
i.e. camera, to form the captured/observed image. The formed image
depends on three main components; (1) The camera (viewer) being
characterized by its projection model (orthographic versus perspective
projection), intrinsic and extrinsic parameters, (2) Illumination (light
source(s)) used to illuminate the object to be captured, characterized
by direction, position, geometry and intensity, and (3) The object
itself being defined by its 3D structure, i.e. shape/surface geometry,
and surface reflectance properties. The formed image at the end is a
two-dimensional pattern of brightness.*

- Analytic appearance models under Lambertian reflectance
- Phenomenological Representation of Arbitrary Reflectance
- Analytic appearance models under arbitrary reflectance
- Appearance models for facial surface reconstruction
- Appearance models for human jaw reconstruction

I started by examining a very restrictive case of convex Lambertian surfaces when image irradiance is captured from a single viewpoint with distant and near illumination assumptions. In such a case, the visible part of the object’s surface constructs the upper hemisphere of the surface normals where spherical harmonics are no longer orthonormal. Assuming no inter-reflections and considering light source(s) distributed uniformly over the upper hemisphere, I proposed the use of hemispherical harmonics (HSH) to model such a case while I proved analytically, and validated experimentally, that the Lambertian reflectance kernel has a more compact harmonic expansion in the hemispherical domain when compared to its spherical counterpart.

*Consider an object at a fixed pose being illuminated by one or
more directional light source(s). Under a single viewpoint, the visible
part will be the hemisphere of surface normals facing the viewer. Thus,
the lighting function, and the surface reflectance, are now defined over
the upper global hemisphere. Given the light distribution, the image
irradiance \(E(\mathbf{n}_j)\) is
generated according to the surface reflectance, then projected onto the
subspace spanned by the image irradiance basis of the given object
[defined by surface normals \(\mathbf{n}_j\) and albedo \(\rho_d(\mathbf{x}_j)\), where the lighting
coefficients lk are the coordinates/weights of the image irradiance in
this subspace. Such basis primarily depends on the reflectance function
basis \(\mathcal{R}_m^n(\mathbf{n}_j)\)
, which is derived according to the Lambertian reflectance. The
‘question’ becomes how to efficiently and accurately represent image
irradiance in this hemispherical domain? Where hemispherical functions
present discontinuities at the boundary of the hemisphere, when
represented in the spherical domain, demanding more coefficients for
accurate representation.*

*From left to right; a graphical representation of the first nine
harmonic expansion coefficients of the Lambertian kernel under distant
illumination, the relative energy maintained by each coefficient and the
cumulative energy. Note that the Lambertian kernel acts as a low-pass
filter, where cut-off frequency in the hemispherical harmonics (HSH)
domain is less than that of the spherical harmonics (SH) and
hemispherical Zernike (HZ) domains, allowing lower number of components
to represent the Lambertian kernel. These values were validated using
Monte-Carlo integration. In the rightmost column, a Lambertian kernel
slice (solid/green) and its first-order approximation using (a) SH
(dashed/blue), (b) HZ (dashed/black), and (c) HSH (dashed/red). Note
that first-order HSH provide good approximation to the Lambertian
kernel.*

Recovering the illumination distribution (i.e., inverse lighting) based on the appearance of objects aids applications such as augmented reality, where the inserted virtual objects are required to share the illumination conditions of the input image. According to previous works, it is common to assume the availability of a prior model of the viewed scene capturing its geometry and albedo. We conducted illumination transfer experiments, where we recovered illumination from an image of an object (referred to as ‘source’) and use the recovered lighting to render a different object (referred to as ‘target’). We quantitatively assessed the proposed image irradiance basis compared with (1) the orthogonalised first-order SH [1], where QR-decomposition is applied to a matrix, whose columns are the first-order SH and (2) the linear combination of higher order SH based on analytical PCA [2].

[1] R. Basri, D. W. Jacobs. “Lambertian reflectance and linear
subspaces”“, IEEE Trans. Pattern Anal. Mach. Intell., 2003, 25, (2),
pp. 218–233

[2] R. Ramamoorthi.”Analytic PCA construction for theoretical analysis
of lighting variability in images of a lambertian object”, IEEE Trans.
Pattern Anal. Mach. Intell., 2002, 24, (10), pp. 1322–1333

*Mean error of the illumination-transferred image when compared
with the groundtruth image as a function of number of distant light
sources used to illuminate 3D surfaces of different objects. Note that
the first-order HSH basis outperforms other basis when transferring
unknown distant illumination.*

**Related publications:**

**Shireen Y. Elhabian** and Aly A. Farag. Towards
Efficient Image Irradiance Modelling of Convex Lambertian Surfaces Under
Single Viewpoint and Frontal Illumination. *IET Computer
Vision*, 7(6) (2013): 478-487.

**Shireen Y. Elhabian**, Ham Rara and Aly Farag. Towards Accurate and
Efficient Representation of Image Irradiance of Convex-Lambertian
Objects under Unknown Near Lighting. *International Conference of
Computer Vision (ICCV)*, pp. 1732 - 1737, Nov 2011.

**Shireen Y. Elhabian**, Ham Rara, Aly Farag. Modeling Lambertian
Surfaces under Unknown Distant Illumination Using Hemispherical
Harmonics. *Eighth Canadian Conference on Computer and Robot
Vision (CRV)*, pp.293-300, 2011.

**Shireen Y. Elhabian**, Ham Rara and Aly Farag. On The Use of
Hemispherical Harmonics for Modeling Images of Object under Known
Distant Illumination. *International Conference of Image
Processing (ICIP)*, pp. 1109 - 1112, Sept 2011.

Moving away from a Lambertian assumption, I realized the need for formulating the image irradiance equation with regard to the local-coordinate frame where illumination is perceived with regard to surface points.

*A point \(\mathbf{x}\) on a
surface patch sees its surrounding world through a unit hemisphere \(\Omega'\) centered at that point. The
surface normal \(\overrightarrow{\mathbf{n}}(\mathbf{x})\)
at the surface point \(\mathbf{x}\)
defines the orientation of the hemisphere. A light ray \(\overrightarrow{\omega}'_i\) incident
to the point \(\mathbf{x}\) is defined
by its direction \((\theta',
\phi')\) in spherical/angular coordinates.*

In particular, I proposed a complete, orthonormal basis to provide a compact and efficient representation for surface bidirectional reflectance distribution function (BRDF), which is defined on the cartesian product of two hemispheres. The proposed basis, which are defined in terms of hemispherical harmonics (HSH), preserve the Helmholtz reciprocity property of BRDFs while avoid the computational complexity inherited from Zernike polynomials that are usually used to construct hemispherical basis.

*Visualization of up-to 3rd order of the proposed Helmholtz
surface reflectance basis at an incident direction \(\overrightarrow{\omega}'_i =
(\pi/3,0)\).*

An analytical as well as experimental justification was presented such that for a given truncating reflectance order, the proposed hemispherical basis provide better approximation accuracy of the BRDF when compared to similar bases in literature. While hemispherical basis provide higher approximation accuracies when compared to spherical ones, basis maintaining helmholtz property was observed to provide higher accuracy levels compared to others.

*Number of orthonormal reflectance basis available up to
reflectance order \(p = r = P\) for (a)
anisotropic and (b) isotropic cases. It can be noted that the proposed
basis has more orthonormal bases at lower orders compared to that of
Koenderink et al. [*] leading to a more compact representation for
surface BRDF.*

[*] J. Koenderink and A. van Doorn. Phenomenlogical description of
bidirectional surface reflection. Journal of the Optical Society of
America, 15(11):2903–2912, 1998.

*The reconstruction of the ideal specular reflectance kernel,
where the light ray (in solid orange) is incident from direction \(\theta'_i = \pi/4\) and \(\phi'_i = 0\) and the perfect specular
direction is plotted in dashed-orange. The approximating series has been
truncated at different orders \(P\) ,
using (a) the proposed HSH-based basis, versus using (b) Zernike-based
basis of [*]. In both cases, the lowest order entails the perfect
diffuse/Lambertian reflectance while the specular lobe begins to be
apparent for orders 1 and higher. Notice the ringing effect due to the
abrupt truncation of the approximating series, however the HSH-based
basis has lower ringing effect compared to the Zernike-based basis
having smaller spurious lobes in directions other than that of the
perfect specular direction, vanishing faster than that of the
Zernike-based basis at higher orders.*

[*] J. Koenderink and A. van Doorn. Phenomenlogical description of
bidirectional surface reflection. Journal of the Optical Society of
America, 15(11):2903–2912, 1998.

The closed form of the proposed basis was presented in case of isotropic and directional hemispherical reflectance. The proposed basis was further validated using scattered reflectance data which might violate the Helmholtz reciprocity property; where such property is maintained in the least-squares sense in the process of fitting the BRDF measurements to the HSH-based basis. The basis was also validated using micro-facet physical BRDF models: Oren-Nayar for diffuse rough surfaces and Torrance-Sparrow and Cook-Torrance for specular rough surfaces. Based on the fact that associated Legendre polynomials are defined for all combinations of polynomial order and degree in contrast to Zernike polynomials, HSH-based basis showed high BRDF approximation accuracy at lower orders.

*The approximation accuracy of CUReT database materials [1] for
different truncating reflectance orders P. The reflectance spectrum for
each material is obtained by projecting randomly drawn CUReT BRDF
measurements using Monte Carlo integration onto the subspace spanned by
the proposed basis in comparison to bases of Westin et al. [2], Gautron
et al. [3], Koenderink et al. [4], Habel and Wimmer [5] and the
Helmholtz basis of Koenderink et al. [4]. Note that BRDF sparse
measurements are interpolated at the drawn samples using the provided
fitted measurements. The sampling points are drawn from the cartesian
product of the incoming and outgoing (hemi)spheres (according to the
basis definition). It can be observed that the proposed basis provides
the highest approximation accuracy for all CUReT, while the spherical
basis [2] provides the lowest accuracy compared to others.*

[1] K. Dana, B. Ginneken, S. Nayar, and J. Koenderink. Reflectance
and texture of real-world surfaces. ACM Transactions on Graphics,
18(1):1–34.

[2] S. Westin, J. Arvo, and K. Torrance. Predicting reflectance
functions from complex surfaces. In Proc. of SIGGRAPH, ACM Press, pages
255–264, 1992.

[3] P. Gautron, J. Krivanek, S.N. Pattanaik, and K. Bouatouch. A novel
hemispherical basis for accurate and efficient rendering. In Proceedings
of the Fifteenth Eurographics conference on Rendering Techniques, pages
321–330. Eurographics Association, 2004.

[4] J. Koenderink and A. van Doorn. Phenomenlogical description of
bidirectional surface reflection. Journal of the Optical Society of
America, 15(11):2903–2912, 1998.

[5] R. Habel and M. Wimmer. Efficient irradiance normal mapping. In
Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D
Graphics and Games, pages 189–195, 2010.

**Related publications:**

**Shireen Y. Elhabian** and Aly A. Farag. Helmholtz
HSH-based Basis: A Compact Phenomenological Representation of Arbitrary
Reflectance, *Informatica*. 2016 Mar 7;26(4): 593-620.

**Shireen Y. Elhabian**, Ham Rara and Aly Farag. Towards
Efficient and Compact Phenomenological Representation of Arbitrary
Bidirectional Surface Reflectance. *British Machine Vision
Conference (BMVC)*, vol. 89, pp 1-11, Aug 2011.

Phenomenological appearance models capture surface appearance through mathematical modeling of the reflection process. Theoretically, the space of all possible images of a fixed-pose object under all possible illumination conditions is infinite dimensional. Nonetheless, due to their low-frequency nature, irradiance signals can be represented using low-order basis functions. Discounting subsurface scattering and surface emittance, this work addressed the question; how to compactly and accurately represent image irradiance under unknown general illumination, given that a surface point sees its surrounding world through the local upper hemisphere oriented by the surface normal at this point.

*The image irradiance equation of an object with isotropic
reflectance.*

In this regard, I formulated the image formation process of isotropic
surfaces under arbitrary distant illumination in the frequency space
while addressing the physical compliance of hemispherical basis for
representing surface reflectance, e.g., Helmholtz reciprocity and
isotropy. The term *irradiance harmonics* was also defined to
enable decoupling illumination and reflectance from the underlying
geometry and pose.

*(a) An object’s surface is illuminated under distant lighting
function defined on the global sphere. (b) An in-depth view of a surface
patch showing surface normals at each surface point. Under the
assumption of non-emitting surfaces, a surface point only sees the
surrounding environment through the local upper hemisphere oriented by
the surface normal at this point, thus the reflection integral can be
defined over the incident local hemisphere where \(\theta'_i \in [0, \pi/2]\). (c) A
zoom-in view at a surface point \(\mathbf{x}\) seeing its surrounding world
through a unit hemisphere \(\Omega'\) centered at the point and
oriented by the surface normal \(\overrightarrow{\mathbf{n}}\) at that
point. Using spherical coordinates, a light ray incident to the point
\(\mathbf{x}\) is defined by its unit
direction \(\overrightarrow{\omega}'_i:
(\theta'_i, \phi'_i\), subtending an incident solid angle
\(d\overrightarrow{\omega}'_i\).
Similarly, an outgoing/reflected light ray is defined by its unit
direction \(\overrightarrow{\omega}'_o:
(\theta'_o, \phi'_o\), subtending an outgoing solid angle
\(d\overrightarrow{\omega}'_o\) .
(d) Visualization of up-to 3rd order of the proposed isotropic Helmholtz
surface reflectance basis at an incident direction \(\overrightarrow{\omega}'_i:
(\pi/3,0)\). The order \(p\)
runs from top to bottom while the associated order \(r\) runs from left to right. The azimuthal
order \(q\), or known as degree, also
runs from left to right for each order \(r\).*

This work provided a closed form of the energy content being maintained by different reflectance modes of the proposed irradiance harmonics. Since specular materials tend to require more basis functions when compared to diffuse ones, the proposed harmonics captures same cumulative energy content, by providing larger number of orthogonal irradiance basis, at lower illumination orders when compared to similar basis in literature.

*Average approximation accuracy of image irradiance as a function
of the illumination order \(n\) based
on CUReT and Merl BRDF databases. Average is computed over the database
materials under the assumption of single distant light source having a
non-decayable constant spectrum. Note that the proposed reflectance
basis maintains higher irradiance approximation accuracy at lower
illumination orders.*

*Sample of frequency-space rendering: Rendered images for the
camel toy from “Weizmann Photometric Stereo Database” with beige-fabric
BRDF under Galileio Tomp (high-frequency) illumination map using the
proposed reflectance basis (red box) versus that of Ramamoorthi and
Hanrahan [1] (blue box) and Nillius and Eklundh [2] (green box) where N
= 12. Path tracing is used to render a unit sphere with the BRDF at the
top-middle as well as the groundtruth rendering at the bottom-left. Mean
absolute error (MAE) is shown below each rendered image where all
intensities are normalized in the range [0,1]. Note that the proposed
basis capture the appearance of the surface reflectance under
high-frequency illumination compared to the others.*

[1] R. Ramamoorthi and P. Hanrahan. Frequency space environment map
rendering. ACM Transactions on Graphics, 21(3):517–526, July 2002.

[2] P. Nillius and J. Eklundh. Phenomenological eigenfunctions for image
irradiance. In Proceedings of the Ninth IEEE International Conference on
Computer Vision - Volume 2, ICCV ’03, pages 568–575, Washington, DC,
USA, 2003. IEEE Computer Society.

This representation was further reduced in dimensionality through analytically deriving the principal components of the image irradiance. In particular, I proposed an analytic formulation for low-dimensional subspace construction in which shading cues lie while preserving the natural structure of an image sample. Thanks to the frequency-space representation of the image irradiance equation where the process of finding such subspace can be cast as establishing a relation between its principal components and that of a deterministic set of basis functions. This resolved the issue of dimensionality since the source of randomness in the imaging process becomes the irradiance harmonics coefficients rather than the whole image realization. Representing images in their natural dimension, i.e. matrices, further lessen the number of parameters to be estimated to define a bilinear projection which maps the image sample to a lower-dimensional bilinear subspace. Since irradiance harmonics enabled decoupling illumination and reflectance from the underlying geometry and pose; this enabled the incorporation of prior information about natural illumination and real world surface materials.

*Subspace construction to model illumination and reflectance:
assuming fixed pose and surface geometry, the frequency space
representation of the image irradiance decouples the image formation
process such that the illumination and reflectance are encoded into the
coefficients \(c^k_s\) of the
irradiance basis functions \(\mathbf{B}_s\) which are geometry and pose
dependent. In case of analytic subspace construction (b), the image
coefficients \(c^k_s\) become the
random variable instead of the image realization. Thus the inherent
curse of dimensionality numerical methods such as Singular Value
Decomposition can be handled.*

A database of natural illumination and real-world surface materials was used to compute a general-purpose basis that captures the full behavior of illumination and reflectance in a lower dimensional subspace for image irradiance representation. Compared to similar bases in the literature, the proposed appearance model achieved higher accuracy levels at lower illumination orders and showed robustness against noise..

*The average reconstruction error for each material in the Merl
database based on the 100 rendered testing images per material where
subspaces were constructed based spherical and hemispherical irradiance
harmonics (refer to the legend) where Q = 98% (percentage of harmonic
variation being maintained when the irradiance harmonics is decomposed).
Note that bilinear subspaces surpass the linear ones while the proposed
HSH-based irradiance harmonics attains minimal reconstruction errors for
all the surface materials.*

*Sample reconstructions of the elephant toy from “Weizmann
Photometric Stereo Database” using the special-walnut BRDF under the
Campus illumination map with different Q(%) using irradiance harmonics
that are based on Zernike-based hemispherical basis deployed by Nillius
and Eklundh [1]. Mean absolute error (MAE) is shown below each rendered
image where all intensities are normalized in the range [0, 1].*

[1] P. Nillius and J. Eklundh. Phenomenological eigenfunctions for image irradiance. In Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2, ICCV ’03, pages 568–575, Washington, DC, USA, 2003. IEEE Computer Society.

*Sample reconstructions of the camel toy from “Weizmann
Photometric Stereo Database” using the derlin BRDF under the Kitchen
illumination map with different Q(%) using irradiance harmonics that are
based on the proposed Helmholtz HSH-based basis. Mean absolute error
(MAE) is shown below each rendered image where all intensities are
normalized in the range [0, 1].*

*Sample reconstructions of the bear toy from “Weizmann Photometric
Stereo Database” using the green-latex BRDF under the Funston Beach
illumination map with different SNR levels using irradiance harmonics
that are based on the proposed Helmholtz HSH-based basis. Mean absolute
error (MAE) is shown below each rendered image where all intensities are
normalized in the range [0, 1].*

**Related publications:**

**Shireen Y. Elhabian** and Aly A. Farag. Image
Irradiance Harmonics: A Phenomenological Model of Image Irradiance of
Arbitrary Surface Reflectance. *IET-Computer Vision*, 8(5),
(2014): 365-381.

**Shireen Y. Elhabian** and Aly Farag. Analytic Bilinear
Appearance Subspace Construction for Modeling Image Irradiance under
Natural Illumination and Non-Lambertian Reflectance. *In Computer
Vision and Pattern Recognition (CVPR)*, 2013 IEEE Conference on,
pp. 1446-1451. IEEE, 2013.

**Shireen Y. Elhabian** and Aly Farag. Modeling Image
Irradiance under Natural Illumination and Isotropic Surface
Reflectance. *19th IEEE International Conference on Image
Processing (ICIP)*, pp. 3013-3016. IEEE, 2012.

I applied the proposed appearance model in a model-based 3D facial shape recovery from a single image of unknown general illumination while accounting explicitly for different human skin types as well as complex illumination conditions. Assuming skin reflectance obeys the Torrance-Sparrow model, I proved analytically that it could be represented by at most a fifth order harmonic basis whose closed form is provided. Experiments on synthetic and real images illustrate the robustness of our appearance model vis-a-vis illumination variation. For more details, visit Statistical shape from shading under natural illumination and arbitrary reflectance.

*A graphical representation of the first thirteen harmonic
expansion coefficients of Torrance Sparrow reflection model under
distant illumination using skin BRDF parameters of Merl Skin BRDF
database. Note that TS acts as a low pass filter, where the 5th order
expansion is sufficient to encodes more than 99% of the BRDF energy
content regardless of the skin type.*

**Related publications:**

**Shireen Y. Elhabian**, Eslam Mostafa, Ham Rara, Aly
Farag. Non-Lambertian
Model-based Facial Shape Recovery from Single Image Under Unknown
General Illumination. *Ninth Canadian Conference on Computer and
Robot Vision (CRV)*, pp. 252-259. IEEE, 2012.

On the medical side, I proposed an appearance-based approach that allows for the reconstruction of plausible human jaw 3D models given a single optical image with unknown illumination. Appearance bases were analytically constructed using the frequency-based representation of the irradiance equation while incorporating prior information about natural illumination and teeth reflectance. Human teeth reflectance was assumed to obey Oren-Nayar model where it is experimentally proved that teeth surface obeys the microfacet theory. The inherent relation between the photometric information and the underlying 3D shape was formulated as a statistical model where the coupled effect of illumination and reflectance is modeled using the Helmholtz hemispherical harmonics-based irradiance harmonics whereas the principal component regression was deployed to carry out the estimation of 3D shapes. I relaxed limiting assumptions of conventional shape-from-shading approaches and was able to reconstruct tooth occlusal surfaces with challenging conditions, such as scattered specular spots and significant changes in color and albedo characteristics resulting from tooth filling. Vis-a-vis dental applications, the results demonstrated a significant increase in accuracy in favor of the proposed approach. For more details, visit Statistical shape from shading under natural illumination and arbitrary reflectance.

*The Oren-Nayar BRDF approximation accuracy of the Helmholtz
HSH-based basis as a function of the approximating order, where in
(left) each curve corresponds to the average approximation accuracy over
all sampled incident directions for several values of surface roughness,
and in (right) it is compared to Zernike-based basis proposed by
Koenderink et al. [1].*

[1] J. Koenderink and A. van Doorn. Phenomenlogical description of bidirectional surface reflection. Journal of the Optical Society of America, 15(11):2903–2912, 1998.

**Related publications:**

**Shireen Y. Elhabian** and Aly A. Farag. Appearance-based
Approach for Complete Human Jaw Shape Reconstruction. *IET
Computer Vision*, 8(5), (2014): 404-418.

**Shireen Y. Elhabian**, Aly Abdelrahim, Aly Farag,
David Tasman, Wael Aboelmaaty, Allan Farman. Clinical Crowns
Shape Reconstruction - an Image-based Approach. *IEEE 10th
International Symposium on Biomedical Imaging (ISBI)*, pp. 93-96.
IEEE, 2013.

Copyright © 2022 Shireen Y. Elhabian. All rights reserved.