Kemmelmacher and Basri.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Experiment Setup\n",
      "The first set of the [USF dataset](http://www.cse.usf.edu/~sarkar/SudeepSarkar/Data_and_Code/Entries/2001/5/8_USF_Human_ID_3-D_Database.html) which contains depth and texture maps of **56** people. Texture maps are normalised by averaging with it's mirror image. The resulting texture map is used as the albedo of the reference model.\n",
      "\n",
      "## Parameters\n",
      "<table>\n",
      "<tr>\n",
      "    <td>Parameter</td>\n",
      "    <td>Value</td>\n",
      "</tr>\n",
      "<tr>\n",
      "    <td>Albedo</td>\n",
      "    <td>Between 0 and 255</td>\n",
      "<tr>\n",
      "<tr>\n",
      "    <td>$\\lambda_1$</td>\n",
      "    <td>110</td>\n",
      "<tr>\n",
      "<tr>\n",
      "    <td>$\\lambda_2$</td>\n",
      "    <td>110</td>\n",
      "<tr>\n",
      "</table>\n",
      "\n",
      "The reference albedo was smooth using a Gaussian ($\\sigma_x = 3, \\sigma_y = 4$). The regularisation terms were similarly smoothed.\n",
      "\n",
      "## Alignment\n",
      "5 reference points are marked on the input image and rigid alignment is performed with the reference model. After alignment images are ``(150, 200)``.\n",
      "\n",
      "# Problem Definition\n",
      "We wish to recover the shape from a single image given a reference shape $(z_{ref})$ and albedo $(\\rho_{ref})$. The image irradiance function is defined as:\n",
      "\n",
      "$$\n",
      "I(x,y) = \\rho (x,y) R(x,y)\n",
      "$$\n",
      "\n",
      "where the reflectance function $R(x,y)$ can be defined in terms of spherical harmonics as:\n",
      "\n",
      "$$\n",
      "R(n(x,y); \\rho (x,y), l) = l^T Y(n(x,y))\n",
      "$$\n",
      "\n",
      "where $n(x,y)$ is the normal at pixel $(x,y)$, $\\rho (x,y)$ the albedo and $l$ the 4-component vector of harmonic expansion coefficients. \n",
      "\n",
      "$Y$ is represents the **first order** spherical harmonic function for each normal, defined in vector form as:\n",
      "\n",
      "$$\n",
      "Y(n) = (1, n_x, n_y, n_z)^T\n",
      "$$\n",
      "\n",
      "In order to preserve discontinuties in the reference model, we define our optimisation problem as in terms of the shape and albedo differences:\n",
      "\n",
      "$$\n",
      "d_z (x,y) = z(x,y) - z_{ref}(x,y) \\\\\n",
      "d_p (x,y) = p(x,y) - p_{ref}(x,y)\n",
      "$$\n",
      "\n",
      "and the optimisation function as:\n",
      "\n",
      "$$\n",
      "\\min_{l,\\rho,z} \\sum_{(x,y) \\in \\Omega} \\left( I(x,y) - \\rho (x,y) l^T Y(n(x,y)) \\right)^2 + \\lambda_1 (\\nabla G \\ast d_z)^2 + \\lambda_2 (\\nabla G \\ast d_p)^2\n",
      "$$\n",
      "\n",
      "We look to solve this optimisation function by seperating the three unknowns, $l$, $\\rho$ and $z$."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Creating a reference albedo map from a known light direction\n",
      "We assume Lambertian reflectance, and thus that the image has been lit by a single dominant light source placed at infinity. Thus, the intensity of the image is defined as:\n",
      "\n",
      "$$\n",
      "I(x,y) = \\rho (x,y) \\max(n(x,y)^T s, 0)\n",
      "$$\n",
      "\n",
      "where $I$ is the pixel intensity, and $s$ is the direction vector of the light. Therefore, given an intensity image, a reference shape and light direction, we can uniquely determine $\\rho (x,y)$ as:\n",
      "\n",
      "$$\n",
      "\\rho (x,y) = \\frac{I(x,y)}{n(x,y)^T s}\n",
      "$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Step 1: Solving for unknown lighting\n",
      "Given that we have a reference shape (and thus normal) and albedo, we can solve for the lighting parameters as follows:\n",
      "$$\n",
      "\\min_{l} \\sum_{(x,y) \\in \\Omega} \\left( I(x,y) - \\rho_{ref}(x,y) l^T \\hat{n}_{ref}(x,y) \\right)^2\n",
      "$$\n",
      "\n",
      "where $\\hat{n} = (1, n_x, n_y, n_z)^T$.\n",
      "\n",
      "Since we only have four unknowns, this represents a highly overdetermined linear least squares problem. We begin by vectorizing the function:\n",
      "\n",
      "$$\n",
      "\\min \\| I - P_{ref} \\circ \\hat{N_{ref}}^T l \\|^2\n",
      "$$\n",
      "\n",
      "where $I$ has been concatenated in to a single column vector $(x_1, y_1, \\ldots, x_N, y_N)$, $P_{ref}$ represents the concatenated column vector of $\\rho_{ref}$ values and $\\hat{N_{ref}}$ is a matrix of the form:\n",
      "\n",
      "$$\n",
      "\\begin{pmatrix}\n",
      "    1 & n^x_1 & n^y_1 & n^z_1 \\\\\n",
      "    \\vdots & \\vdots & \\vdots & \\vdots \\\\\n",
      "    1 & n^x_N & n^y_N & n^z_N\n",
      "\\end{pmatrix}\n",
      "$$\n",
      "\n",
      "Now, this optimisation function is minimised when $\\frac{\\partial f}{\\partial l} = 0$. Therefore, by taking the derivative, we arrive at the solution:\n",
      "\n",
      "$$\n",
      "I - (P \\circ \\hat{N} l)(p \\circ \\hat{N}^T) = 0 \\\\\n",
      "P \\circ \\hat{N}^T I - P \\circ \\hat{N}^T P \\circ \\hat{N} l = 0 \\\\\n",
      "l = \\left[(P \\circ \\hat{N})^T (P \\circ \\hat{N}) \\right]^{-1} (P \\circ \\hat{N})^T I\n",
      "$$\n",
      "\n",
      "Given that our system is overdetermined, we note that $\\left[(P \\circ \\hat{N})^T (P \\circ \\hat{N}) \\right]^{-1} (P \\circ \\hat{N})^T$ is equivalent to the pseudo-inverse of $P \\circ \\hat{N}$."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}