Appearance Modeling

Motivation: Various vision tasks are usually confronted by appearance variations due to changes in illumination. For instance, in a recognition system, it has been shown that the variability in the appearance of human face is dominated by changes to lighting conditions rather than the individual’s identity. My passion encompassed the extraction of information about the contents of an image based on an understanding of the underlying physics that governs how the image was formed. Theoretically, due to the arbitrariness of the lighting function, the space of all possible images of a fixed-pose object under all possible illumination conditions has an infinite dimension. Nonetheless, it has been proven that the set of images of a convex Lambertian surface under distant illumination lies near a low-dimensional linear subspace. This result was also extended to include non-Lambertian objects with non-convex geometry. As such, vision applications, concerned with the recovery of illumination, reflectance, and/or surface geometry from images, would benefit from a low-dimensional generative model that captures appearance variations with regard to illumination conditions and surface reflectance properties.

In my Ph.D. research, I proposed an analytic formulation of appearance subspace construction to capture the full behavior of complex illumination and real-world reflectance, and in the process accounted for the hemispherical nature of surface reflectance functions to model physically plausible surface materials. The essence of my work in this area is phenomenological reflectance models that capture appearance variations through mathematical abstraction of the reflection process. In particular, the harmonic expansion of the image irradiance equation can be used to derive an analytic subspace to represent images under fixed-pose, but different illumination conditions.

Image formation triology, light source(s) are used to illuminate a 3D object in space which will be captured by an imaging system, i.e. camera, to form the captured/observed image. The formed image depends on three main components; (1) The camera (viewer) being characterized by its projection model (orthographic versus perspective projection), intrinsic and extrinsic parameters, (2) Illumination (light source(s)) used to illuminate the object to be captured, characterized by direction, position, geometry and intensity, and (3) The object itself being defined by its 3D structure, i.e. shape/surface geometry, and surface reflectance properties. The formed image at the end is a two-dimensional pattern of brightness.

Analytic appearance models under Lambertian reflectance

I started by examining a very restrictive case of convex Lambertian surfaces when image irradiance is captured from a single viewpoint with distant and near illumination assumptions. In such a case, the visible part of the object’s surface constructs the upper hemisphere of the surface normals where spherical harmonics are no longer orthonormal. Assuming no inter-reflections and considering light source(s) distributed uniformly over the upper hemisphere, I proposed the use of hemispherical harmonics (HSH) to model such a case while I proved analytically, and validated experimentally, that the Lambertian reflectance kernel has a more compact harmonic expansion in the hemispherical domain when compared to its spherical counterpart.

Consider an object at a fixed pose being illuminated by one or more directional light source(s). Under a single viewpoint, the visible part will be the hemisphere of surface normals facing the viewer. Thus, the lighting function, and the surface reflectance, are now defined over the upper global hemisphere. Given the light distribution, the image irradiance \(E(\mathbf{n}_j)\) is generated according to the surface reflectance, then projected onto the subspace spanned by the image irradiance basis of the given object [defined by surface normals \(\mathbf{n}_j\) and albedo \(\rho_d(\mathbf{x}_j)\), where the lighting coefficients lk are the coordinates/weights of the image irradiance in this subspace. Such basis primarily depends on the reflectance function basis \(\mathcal{R}_m^n(\mathbf{n}_j)\) , which is derived according to the Lambertian reflectance. The ‘question’ becomes how to efficiently and accurately represent image irradiance in this hemispherical domain? Where hemispherical functions present discontinuities at the boundary of the hemisphere, when represented in the spherical domain, demanding more coefficients for accurate representation.

From left to right; a graphical representation of the first nine harmonic expansion coefficients of the Lambertian kernel under distant illumination, the relative energy maintained by each coefficient and the cumulative energy. Note that the Lambertian kernel acts as a low-pass filter, where cut-off frequency in the hemispherical harmonics (HSH) domain is less than that of the spherical harmonics (SH) and hemispherical Zernike (HZ) domains, allowing lower number of components to represent the Lambertian kernel. These values were validated using Monte-Carlo integration. In the rightmost column, a Lambertian kernel slice (solid/green) and its first-order approximation using (a) SH (dashed/blue), (b) HZ (dashed/black), and (c) HSH (dashed/red). Note that first-order HSH provide good approximation to the Lambertian kernel.

Recovering the illumination distribution (i.e., inverse lighting) based on the appearance of objects aids applications such as augmented reality, where the inserted virtual objects are required to share the illumination conditions of the input image. According to previous works, it is common to assume the availability of a prior model of the viewed scene capturing its geometry and albedo. We conducted illumination transfer experiments, where we recovered illumination from an image of an object (referred to as ‘source’) and use the recovered lighting to render a different object (referred to as ‘target’). We quantitatively assessed the proposed image irradiance basis compared with (1) the orthogonalised first-order SH [1], where QR-decomposition is applied to a matrix, whose columns are the first-order SH and (2) the linear combination of higher order SH based on analytical PCA [2].

[1] R. Basri, D. W. Jacobs. “Lambertian reflectance and linear subspaces”“, IEEE Trans. Pattern Anal. Mach. Intell., 2003, 25, (2), pp. 218–233
[2] R. Ramamoorthi.”Analytic PCA construction for theoretical analysis of lighting variability in images of a lambertian object”, IEEE Trans. Pattern Anal. Mach. Intell., 2002, 24, (10), pp. 1322–1333

Mean error of the illumination-transferred image when compared with the groundtruth image as a function of number of distant light sources used to illuminate 3D surfaces of different objects. Note that the first-order HSH basis outperforms other basis when transferring unknown distant illumination.

Related publications:

Shireen Y. Elhabian and Aly A. Farag. Towards Efficient Image Irradiance Modelling of Convex Lambertian Surfaces Under Single Viewpoint and Frontal Illumination. IET Computer Vision, 7(6) (2013): 478-487.

Shireen Y. Elhabian, Ham Rara and Aly Farag. Towards Accurate and Efficient Representation of Image Irradiance of Convex-Lambertian Objects under Unknown Near Lighting. International Conference of Computer Vision (ICCV), pp. 1732 - 1737, Nov 2011.

Shireen Y. Elhabian, Ham Rara, Aly Farag. Modeling Lambertian Surfaces under Unknown Distant Illumination Using Hemispherical Harmonics. Eighth Canadian Conference on Computer and Robot Vision (CRV), pp.293-300, 2011.

Shireen Y. Elhabian, Ham Rara and Aly Farag. On The Use of Hemispherical Harmonics for Modeling Images of Object under Known Distant Illumination. International Conference of Image Processing (ICIP), pp. 1109 - 1112, Sept 2011.

Back to top

Phenomenological Representation of Arbitrary Reflectance

Moving away from a Lambertian assumption, I realized the need for formulating the image irradiance equation with regard to the local-coordinate frame where illumination is perceived with regard to surface points.

A point \(\mathbf{x}\) on a surface patch sees its surrounding world through a unit hemisphere \(\Omega'\) centered at that point. The surface normal \(\overrightarrow{\mathbf{n}}(\mathbf{x})\) at the surface point \(\mathbf{x}\) defines the orientation of the hemisphere. A light ray \(\overrightarrow{\omega}'_i\) incident to the point \(\mathbf{x}\) is defined by its direction \((\theta', \phi')\) in spherical/angular coordinates.

In particular, I proposed a complete, orthonormal basis to provide a compact and efficient representation for surface bidirectional reflectance distribution function (BRDF), which is defined on the cartesian product of two hemispheres. The proposed basis, which are defined in terms of hemispherical harmonics (HSH), preserve the Helmholtz reciprocity property of BRDFs while avoid the computational complexity inherited from Zernike polynomials that are usually used to construct hemispherical basis.

Visualization of up-to 3rd order of the proposed Helmholtz surface reflectance basis at an incident direction \(\overrightarrow{\omega}'_i = (\pi/3,0)\).

An analytical as well as experimental justification was presented such that for a given truncating reflectance order, the proposed hemispherical basis provide better approximation accuracy of the BRDF when compared to similar bases in literature. While hemispherical basis provide higher approximation accuracies when compared to spherical ones, basis maintaining helmholtz property was observed to provide higher accuracy levels compared to others.

Number of orthonormal reflectance basis available up to reflectance order \(p = r = P\) for (a) anisotropic and (b) isotropic cases. It can be noted that the proposed basis has more orthonormal bases at lower orders compared to that of Koenderink et al. [*] leading to a more compact representation for surface BRDF.
[*] J. Koenderink and A. van Doorn. Phenomenlogical description of bidirectional surface reflection. Journal of the Optical Society of America, 15(11):2903–2912, 1998.

The reconstruction of the ideal specular reflectance kernel, where the light ray (in solid orange) is incident from direction \(\theta'_i = \pi/4\) and \(\phi'_i = 0\) and the perfect specular direction is plotted in dashed-orange. The approximating series has been truncated at different orders \(P\) , using (a) the proposed HSH-based basis, versus using (b) Zernike-based basis of [*]. In both cases, the lowest order entails the perfect diffuse/Lambertian reflectance while the specular lobe begins to be apparent for orders 1 and higher. Notice the ringing effect due to the abrupt truncation of the approximating series, however the HSH-based basis has lower ringing effect compared to the Zernike-based basis having smaller spurious lobes in directions other than that of the perfect specular direction, vanishing faster than that of the Zernike-based basis at higher orders.
[*] J. Koenderink and A. van Doorn. Phenomenlogical description of bidirectional surface reflection. Journal of the Optical Society of America, 15(11):2903–2912, 1998.

The closed form of the proposed basis was presented in case of isotropic and directional hemispherical reflectance. The proposed basis was further validated using scattered reflectance data which might violate the Helmholtz reciprocity property; where such property is maintained in the least-squares sense in the process of fitting the BRDF measurements to the HSH-based basis. The basis was also validated using micro-facet physical BRDF models: Oren-Nayar for diffuse rough surfaces and Torrance-Sparrow and Cook-Torrance for specular rough surfaces. Based on the fact that associated Legendre polynomials are defined for all combinations of polynomial order and degree in contrast to Zernike polynomials, HSH-based basis showed high BRDF approximation accuracy at lower orders.

The approximation accuracy of CUReT database materials [1] for different truncating reflectance orders P. The reflectance spectrum for each material is obtained by projecting randomly drawn CUReT BRDF measurements using Monte Carlo integration onto the subspace spanned by the proposed basis in comparison to bases of Westin et al. [2], Gautron et al. [3], Koenderink et al. [4], Habel and Wimmer [5] and the Helmholtz basis of Koenderink et al. [4]. Note that BRDF sparse measurements are interpolated at the drawn samples using the provided fitted measurements. The sampling points are drawn from the cartesian product of the incoming and outgoing (hemi)spheres (according to the basis definition). It can be observed that the proposed basis provides the highest approximation accuracy for all CUReT, while the spherical basis [2] provides the lowest accuracy compared to others.

[1] K. Dana, B. Ginneken, S. Nayar, and J. Koenderink. Reflectance and texture of real-world surfaces. ACM Transactions on Graphics, 18(1):1–34.
[2] S. Westin, J. Arvo, and K. Torrance. Predicting reflectance functions from complex surfaces. In Proc. of SIGGRAPH, ACM Press, pages 255–264, 1992.
[3] P. Gautron, J. Krivanek, S.N. Pattanaik, and K. Bouatouch. A novel hemispherical basis for accurate and efficient rendering. In Proceedings of the Fifteenth Eurographics conference on Rendering Techniques, pages 321–330. Eurographics Association, 2004.
[4] J. Koenderink and A. van Doorn. Phenomenlogical description of bidirectional surface reflection. Journal of the Optical Society of America, 15(11):2903–2912, 1998.
[5] R. Habel and M. Wimmer. Efficient irradiance normal mapping. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games, pages 189–195, 2010.

Related publications:

Shireen Y. Elhabian and Aly A. Farag. Helmholtz HSH-based Basis: A Compact Phenomenological Representation of Arbitrary Reflectance, Informatica. 2016 Mar 7;26(4): 593-620.

Shireen Y. Elhabian, Ham Rara and Aly Farag. Towards Efficient and Compact Phenomenological Representation of Arbitrary Bidirectional Surface Reflectance. British Machine Vision Conference (BMVC), vol. 89, pp 1-11, Aug 2011.

Back to top

Analytic appearance models under arbitrary reflectance

Phenomenological appearance models capture surface appearance through mathematical modeling of the reflection process. Theoretically, the space of all possible images of a fixed-pose object under all possible illumination conditions is infinite dimensional. Nonetheless, due to their low-frequency nature, irradiance signals can be represented using low-order basis functions. Discounting subsurface scattering and surface emittance, this work addressed the question; how to compactly and accurately represent image irradiance under unknown general illumination, given that a surface point sees its surrounding world through the local upper hemisphere oriented by the surface normal at this point.

The image irradiance equation of an object with isotropic reflectance.

In this regard, I formulated the image formation process of isotropic surfaces under arbitrary distant illumination in the frequency space while addressing the physical compliance of hemispherical basis for representing surface reflectance, e.g., Helmholtz reciprocity and isotropy. The term irradiance harmonics was also defined to enable decoupling illumination and reflectance from the underlying geometry and pose.

(a) An object’s surface is illuminated under distant lighting function defined on the global sphere. (b) An in-depth view of a surface patch showing surface normals at each surface point. Under the assumption of non-emitting surfaces, a surface point only sees the surrounding environment through the local upper hemisphere oriented by the surface normal at this point, thus the reflection integral can be defined over the incident local hemisphere where \(\theta'_i \in [0, \pi/2]\). (c) A zoom-in view at a surface point \(\mathbf{x}\) seeing its surrounding world through a unit hemisphere \(\Omega'\) centered at the point and oriented by the surface normal \(\overrightarrow{\mathbf{n}}\) at that point. Using spherical coordinates, a light ray incident to the point \(\mathbf{x}\) is defined by its unit direction \(\overrightarrow{\omega}'_i: (\theta'_i, \phi'_i\), subtending an incident solid angle \(d\overrightarrow{\omega}'_i\). Similarly, an outgoing/reflected light ray is defined by its unit direction \(\overrightarrow{\omega}'_o: (\theta'_o, \phi'_o\), subtending an outgoing solid angle \(d\overrightarrow{\omega}'_o\) . (d) Visualization of up-to 3rd order of the proposed isotropic Helmholtz surface reflectance basis at an incident direction \(\overrightarrow{\omega}'_i: (\pi/3,0)\). The order \(p\) runs from top to bottom while the associated order \(r\) runs from left to right. The azimuthal order \(q\), or known as degree, also runs from left to right for each order \(r\).

This work provided a closed form of the energy content being maintained by different reflectance modes of the proposed irradiance harmonics. Since specular materials tend to require more basis functions when compared to diffuse ones, the proposed harmonics captures same cumulative energy content, by providing larger number of orthogonal irradiance basis, at lower illumination orders when compared to similar basis in literature.

Average approximation accuracy of image irradiance as a function of the illumination order \(n\) based on CUReT and Merl BRDF databases. Average is computed over the database materials under the assumption of single distant light source having a non-decayable constant spectrum. Note that the proposed reflectance basis maintains higher irradiance approximation accuracy at lower illumination orders.

Sample of frequency-space rendering: Rendered images for the camel toy from “Weizmann Photometric Stereo Database” with beige-fabric BRDF under Galileio Tomp (high-frequency) illumination map using the proposed reflectance basis (red box) versus that of Ramamoorthi and Hanrahan [1] (blue box) and Nillius and Eklundh [2] (green box) where N = 12. Path tracing is used to render a unit sphere with the BRDF at the top-middle as well as the groundtruth rendering at the bottom-left. Mean absolute error (MAE) is shown below each rendered image where all intensities are normalized in the range [0,1]. Note that the proposed basis capture the appearance of the surface reflectance under high-frequency illumination compared to the others.

[1] R. Ramamoorthi and P. Hanrahan. Frequency space environment map rendering. ACM Transactions on Graphics, 21(3):517–526, July 2002.
[2] P. Nillius and J. Eklundh. Phenomenological eigenfunctions for image irradiance. In Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2, ICCV ’03, pages 568–575, Washington, DC, USA, 2003. IEEE Computer Society.

This representation was further reduced in dimensionality through analytically deriving the principal components of the image irradiance. In particular, I proposed an analytic formulation for low-dimensional subspace construction in which shading cues lie while preserving the natural structure of an image sample. Thanks to the frequency-space representation of the image irradiance equation where the process of finding such subspace can be cast as establishing a relation between its principal components and that of a deterministic set of basis functions. This resolved the issue of dimensionality since the source of randomness in the imaging process becomes the irradiance harmonics coefficients rather than the whole image realization. Representing images in their natural dimension, i.e. matrices, further lessen the number of parameters to be estimated to define a bilinear projection which maps the image sample to a lower-dimensional bilinear subspace. Since irradiance harmonics enabled decoupling illumination and reflectance from the underlying geometry and pose; this enabled the incorporation of prior information about natural illumination and real world surface materials.

Subspace construction to model illumination and reflectance: assuming fixed pose and surface geometry, the frequency space representation of the image irradiance decouples the image formation process such that the illumination and reflectance are encoded into the coefficients \(c^k_s\) of the irradiance basis functions \(\mathbf{B}_s\) which are geometry and pose dependent. In case of analytic subspace construction (b), the image coefficients \(c^k_s\) become the random variable instead of the image realization. Thus the inherent curse of dimensionality numerical methods such as Singular Value Decomposition can be handled.

A database of natural illumination and real-world surface materials was used to compute a general-purpose basis that captures the full behavior of illumination and reflectance in a lower dimensional subspace for image irradiance representation. Compared to similar bases in the literature, the proposed appearance model achieved higher accuracy levels at lower illumination orders and showed robustness against noise..

The average reconstruction error for each material in the Merl database based on the 100 rendered testing images per material where subspaces were constructed based spherical and hemispherical irradiance harmonics (refer to the legend) where Q = 98% (percentage of harmonic variation being maintained when the irradiance harmonics is decomposed). Note that bilinear subspaces surpass the linear ones while the proposed HSH-based irradiance harmonics attains minimal reconstruction errors for all the surface materials.

Sample reconstructions of the elephant toy from “Weizmann Photometric Stereo Database” using the special-walnut BRDF under the Campus illumination map with different Q(%) using irradiance harmonics that are based on Zernike-based hemispherical basis deployed by Nillius and Eklundh [1]. Mean absolute error (MAE) is shown below each rendered image where all intensities are normalized in the range [0, 1].

[1] P. Nillius and J. Eklundh. Phenomenological eigenfunctions for image irradiance. In Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2, ICCV ’03, pages 568–575, Washington, DC, USA, 2003. IEEE Computer Society.

Sample reconstructions of the camel toy from “Weizmann Photometric Stereo Database” using the derlin BRDF under the Kitchen illumination map with different Q(%) using irradiance harmonics that are based on the proposed Helmholtz HSH-based basis. Mean absolute error (MAE) is shown below each rendered image where all intensities are normalized in the range [0, 1].

Sample reconstructions of the bear toy from “Weizmann Photometric Stereo Database” using the green-latex BRDF under the Funston Beach illumination map with different SNR levels using irradiance harmonics that are based on the proposed Helmholtz HSH-based basis. Mean absolute error (MAE) is shown below each rendered image where all intensities are normalized in the range [0, 1].

Related publications:

Shireen Y. Elhabian and Aly A. Farag. Image Irradiance Harmonics: A Phenomenological Model of Image Irradiance of Arbitrary Surface Reflectance. IET-Computer Vision, 8(5), (2014): 365-381.

Shireen Y. Elhabian and Aly Farag. Analytic Bilinear Appearance Subspace Construction for Modeling Image Irradiance under Natural Illumination and Non-Lambertian Reflectance. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pp. 1446-1451. IEEE, 2013.

Shireen Y. Elhabian and Aly Farag. Modeling Image Irradiance under Natural Illumination and Isotropic Surface Reflectance. 19th IEEE International Conference on Image Processing (ICIP), pp. 3013-3016. IEEE, 2012.

Back to top

Appearance models for facial surface reconstruction

I applied the proposed appearance model in a model-based 3D facial shape recovery from a single image of unknown general illumination while accounting explicitly for different human skin types as well as complex illumination conditions. Assuming skin reflectance obeys the Torrance-Sparrow model, I proved analytically that it could be represented by at most a fifth order harmonic basis whose closed form is provided. Experiments on synthetic and real images illustrate the robustness of our appearance model vis-a-vis illumination variation. For more details, visit Statistical shape from shading under natural illumination and arbitrary reflectance.

A graphical representation of the first thirteen harmonic expansion coefficients of Torrance Sparrow reflection model under distant illumination using skin BRDF parameters of Merl Skin BRDF database. Note that TS acts as a low pass filter, where the 5th order expansion is sufficient to encodes more than 99% of the BRDF energy content regardless of the skin type.

Related publications:

Shireen Y. Elhabian, Eslam Mostafa, Ham Rara, Aly Farag. Non-Lambertian Model-based Facial Shape Recovery from Single Image Under Unknown General Illumination. Ninth Canadian Conference on Computer and Robot Vision (CRV), pp. 252-259. IEEE, 2012.

Back to top

Appearance models for human jaw reconstruction

On the medical side, I proposed an appearance-based approach that allows for the reconstruction of plausible human jaw 3D models given a single optical image with unknown illumination. Appearance bases were analytically constructed using the frequency-based representation of the irradiance equation while incorporating prior information about natural illumination and teeth reflectance. Human teeth reflectance was assumed to obey Oren-Nayar model where it is experimentally proved that teeth surface obeys the microfacet theory. The inherent relation between the photometric information and the underlying 3D shape was formulated as a statistical model where the coupled effect of illumination and reflectance is modeled using the Helmholtz hemispherical harmonics-based irradiance harmonics whereas the principal component regression was deployed to carry out the estimation of 3D shapes. I relaxed limiting assumptions of conventional shape-from-shading approaches and was able to reconstruct tooth occlusal surfaces with challenging conditions, such as scattered specular spots and significant changes in color and albedo characteristics resulting from tooth filling. Vis-a-vis dental applications, the results demonstrated a significant increase in accuracy in favor of the proposed approach. For more details, visit Statistical shape from shading under natural illumination and arbitrary reflectance.

The Oren-Nayar BRDF approximation accuracy of the Helmholtz HSH-based basis as a function of the approximating order, where in (left) each curve corresponds to the average approximation accuracy over all sampled incident directions for several values of surface roughness, and in (right) it is compared to Zernike-based basis proposed by Koenderink et al. [1].

[1] J. Koenderink and A. van Doorn. Phenomenlogical description of bidirectional surface reflection. Journal of the Optical Society of America, 15(11):2903–2912, 1998.

Related publications:

Shireen Y. Elhabian and Aly A. Farag. Appearance-based Approach for Complete Human Jaw Shape Reconstruction. IET Computer Vision, 8(5), (2014): 404-418.

Shireen Y. Elhabian, Aly Abdelrahim, Aly Farag, David Tasman, Wael Aboelmaaty, Allan Farman. Clinical Crowns Shape Reconstruction - an Image-based Approach. IEEE 10th International Symposium on Biomedical Imaging (ISBI), pp. 93-96. IEEE, 2013.

Back to top