.. vim: set fileencoding=utf-8 :
.. Manuel Guenther <manuek.guether@idiap.ch>
.. Mon Aug 25 19:15:29 CEST 2014

==========================================
 Running the Face Recognition Experiments
==========================================

.. testsetup:: *

  from __future__ import print_function
  import numpy
  import scipy.spatial
  import bob.db.atnt
  import bob.io.base
  import bob.io.image
  import bob.ip.base
  import bob.ip.gabor
  import bob.learn.linear
  import bob.measure

  training_features = [(numpy.random.rand(400) * 255) for i in range(20)]
  probe_features = [(numpy.random.rand(400) * 255) for i in range(5)]
  model_features = {1: [(numpy.random.rand(400) * 255) for i in range(5)]}
  models = [[(numpy.random.rand(400) * 255) for i in range(5)]]
  training_image = numpy.random.rand(20,20) * 255
  model_image = numpy.random.rand(20,20) * 255
  probe_image = numpy.random.rand(20,20) * 255

  positives = numpy.random.rand(100)
  negatives = numpy.random.rand(500) - 0.5

To run the examples, just call the scripts from within the ``bin`` directory, e.g.:

.. code-block:: sh

  $ ./bin/eigenface.py

If you installed the image database in another folder than ``Database``, please give this directory as parameter to the script, e.g.:

.. code-block:: sh

  $ ./bin/eigenface.py <AT&T_DATABASE_DIR>

There are three experiment scripts:

.. code-block:: sh

  $ ./bin/eigenface.py
  $ ./bin/gabor_graph.py
  $ ./bin/dct_ubm.py

that perform more or less complicated face verification experiments using an *unbiased* evaluation protocol.
Each experiment creates an ROC curve that contains the final verification result of the test.
The generated files will be ``eigenface.pdf``, ``gabor_graph.pdf`` and ``dct_ubm.pdf``, respectively.

Since the complexity of the algorithms increase, the expected execution time of them differ a lot.
While the eigenface experiment should be finished in a couple of seconds, the Gabor graph experiment could take some minutes, and the UBM/GMM model needs in the order of half an hour to compute.

.. note::

  The example code that is presented here differ slightly from the code in the source files.
  Here, only the concepts of the functions should be clarified, while the source files contain code that is better arranged and computes faster.


In this small example, we use the :ref:`bob.db.atnt <bob.db.atnt>` database.
This database is compatible with all other :py:class:`bob.db.verification.utils.Database`\s, so any other :ref:`verification_databases` can be used in a direct replacement of :ref:`bob.db.atnt <bob.db.atnt>`.


The Eigenface Example
~~~~~~~~~~~~~~~~~~~~~
The eigenface example follows the work-flow that is presented in the original paper *Eigenfaces for Recognition* [TP91]_ by Turk and Pentland.
First, it creates an :py:class:`bob.db.atnt.Database` object to query the :ref:`bob.db.atnt <bob.db.atnt>` database:

.. doctest::

  >>> atnt_db = bob.db.atnt.Database()

For training the projection matrix, the training images (which are in gray-scale) are be read using the generic :py:func:`bob.io.base.load` function, which returns the data as a :py:class:`numpy.ndarray`:

.. doctest::

  >>> training_image_files = atnt_db.objects(groups = 'world')
  >>> for training_file in training_image_files:
  ...   # load image
  ...   training_image = bob.io.base.load(training_file.make_path(...)) #doctest:+SKIP
  ...   # linearize pixels
  ...   training_feature = training_image.flatten()

Since the face images in the AT&T database are already cropped, they can simply be linearized (converted into one long vector) and put into a 2D array with one sample in each row, and no :py:class:`bob.ip.base.FaceEyesNorm` needs to be applied to align the faces using the :py:meth:`bob.db.verification.utils.Database.annotations` of the eyes:

.. doctest::

  >>> training_set = numpy.vstack(training_features)

which is then used to train a :py:class:`bob.learn.linear.Machine` using a :py:class:`bob.learn.linear.PCATrainer`:

.. doctest::

  >>> pca_trainer = bob.learn.linear.PCATrainer()
  >>> pca_machine, eigen_values = pca_trainer.train(training_set)

For some distance functions, the eigenvalues are needed, but in our example we just ignore them.

After training, the model and probe images are loaded, linearized, and projected into the eigenspace using the trained ``pca_machine``:

.. doctest::

  >>> model_image_files = atnt_db.objects(groups = 'dev', purposes = 'enroll')
  >>> for model_file in model_image_files:
  ...   # load image
  ...   model_image = bob.io.base.load(model_file.make_path(...)) #doctest:+SKIP
  ...   # project to PCA subspace
  ...   model_feature = pca_machine.forward(model_image.flatten())

  >>> probe_image_files = atnt_db.objects(groups = 'dev', purposes = 'probe')
  >>> for probe_file in probe_image_files:
  ...   # load image
  ...   probe_image = bob.io.base.load(probe_file.make_path(...)) #doctest:+SKIP
  ...   # project to PCA subspace
  ...   model_feature = pca_machine.forward(probe_image.flatten())

To follow the evaluation protocol, we *enroll* a client model for each client, simply by collecting all model feature vectors:

.. doctest::

  >>> model_ids = [client.id for client in atnt_db.clients(groups = 'dev')]
  >>> for model_image_id in model_features:
  ...   # query the database for the model id of the current file id
  ...   model_id = atnt_db.get_client_id_from_file_id(model_image_id)
  ...   # append feature for the current model id
  ...   models[model_id].append(model_features[model_feature_id]) #doctest:+SKIP


To compute the verification result, each model feature is compared to each probe feature by computing the :py:func:`scipy.spatial.distance.euclidean` distance:

.. doctest::

  >>> for model in models:
  ...  for probe_feature in probe_features:
  ...    for model_feature in model:
  ...      score = scipy.spatial.distance.euclidean(model_feature, probe_feature)

Finally, all scores of one model and one probe are averaged to get the final score for this pair.

The results are divided into a list of positive scores (model and probe are from the same identity) and a a list of negative scores (identities of model and probe differ).
Using these lists, the ROC curve is plotted using functionality from :ref:`bob.measure <bob.measure>`:

.. doctest::

  >>> bob.measure.plot.roc(negatives, positives) #doctest:+ELLIPSIS
  [...]

.. image:: eigenface.png
  :scale: 100 %

and the performance is computed:

.. doctest::

  >>> threshold = bob.measure.eer_threshold(negatives, positives)
  >>> FAR, FRR = bob.measure.farfrr(negatives, positives, threshold)

The expected result is: FAR 9.15% and FRR 9% at threshold -9276.2

.. note::

  Here we plot the ROC curves with logarithmic FAR axis --- to highlight the interesting part of the curve, i.e., where the FAR values are small.

.. note::

  Computing eigenfaces with such a low amount of training data is usually not an excellent idea.
  Hence, the performance in this example is relatively poor.


Gabor jet comparisons
~~~~~~~~~~~~~~~~~~~~~
A better face verification example uses Gabor jet features [WFKM97]_ .
In this example we do not define a face graph, but instead we use the :py:class:`bob.ip.gabor.Jet`\s at several grid positions in the image.
To do that, we define:

.. doctest::

  >>> graph = bob.ip.gabor.Graph((8,6), (104,86), (4,4))

that will create Gabor graphs with node positions from (8,6) to (104,86) with step size (4,4).

.. note::

  The resolution of the images in the AT&T database is 92x112.
  Of course, there are ways to automatically get the size of the images, but for brevity we hard-coded the resolution of the images.

.. note::

  The Gabor graph extraction does not require a training stage.
  Therefore, in opposition to the eigenface example, the training images are not used in this example.

Now, the Gabor graph features can be extracted from the model and probe images.
Here is the code for the model graphs only:

.. doctest::

  >>> gabor_wavelet_transform = bob.ip.gabor.Transform()
  >>> model_image_files = atnt_db.objects(groups = 'dev', purposes = 'enroll')
  >>> for model_file in model_image_files:
  ...   # load image
  ...   model_image = bob.io.base.load(model_file.make_path(...)) #doctest:+SKIP
  ...   # create the Gabor transformed image
  ...   trafo_image = gabor_wavelet_transform.transform(model_image)
  ...   model_feature = graph.extract(model_image)

For model enrollment, as above we simply collect all enrollment features.
To compare the Gabor graphs, several methods can be applied.
Again, many choices for the Gabor jet comparison exist, here we take a novel Gabor phase based :py:class:`bob.ip.gabor.Similarity` function [GHW12]_:

.. doctest::

  >>> similarity_function = bob.ip.gabor.Similarity("PhaseDiffPlusCanberra", gabor_wavelet_transform)

Since we have several local features, we can exploit this fact.
For each local position, we compute the similarity between the probe feature at this position and all model features and take the maximum value:

.. code-block:: python

  >>> for model_id in model_ids:
  ...  for probe_feature in probe_features:
  ...    for model_feature in models[model_id]:
  ...      for node_index in range(len(probe_feature)):
  ...        scores[...] = similarity_function.similarity(model_feature[node_index], probe_feature[node_index])
  ...    score = numpy.average(numpy.max(scores, axis = 0))

The evaluation is identical to the evaluation in the eigenface example.
Since this method is better for suited for small image databases, the resulting verification rates are better.
The expected ROC curve is:

.. image:: gabor_graph.png
  :scale: 100 %

while the expected verification result is: FAR 3% and FRR 3% at distance threshold 0.5912


The UBM/GMM modeling of DCT Blocks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The last example shows a quite complicated, but successful algorithm.
The first step is the feature extraction of the training image features and the collection of them in a 2D array.
In this experiment we will use *Discrete Cosine Transform* (DCT) block features using :py:class:`bob.ip.base.DCTFeatures` [MM09]_:

.. doctest::

  >>> dct_extractor = bob.ip.base.DCTFeatures(45, (12, 12), (11, 11))
  >>> training_image_files = atnt_db.objects(groups = 'world')
  >>> for training_file in training_image_files:
  ...   # load training image
  ...   training_image = bob.io.base.load(training_file.make_path(...))  #doctest:+SKIP
  ...   # extract DCT block features
  ...   training_features = dct_extractor(training_image)

Hence, from every image, several DCT block features are extracted independently.
All these features are mixed together to build the training set:

.. code-block:: python

  >>> training_set = numpy.vstack(training_features_list)

With these training features, a *universal background model* (UBM) is computed [RQD00]_.
It is a *Gaussian Mixture Model* (GMM) that holds information about the overall distribution of DCT features in facial images.
The UBM model is trained using a :py:class:`bob.learn.misc.KMeansTrainer` to estimate the means of the Gaussians:

.. code-block:: python

  >>> kmeans_machine = bob.learn.misc.KMeansMachine(...)
  >>> kmeans_trainer = bob.learn.misc.KMeansTrainer()
  >>> kmeans_trainer.train(kmeans, training_set)

Afterward, the UBM is initialized with the results of the k-means training:

.. code-block:: python

  >>> ubm = bob.learn.misc.GMMMachine(...)
  >>> ubm.means = kmeans_machine.means
  >>> [variances, weights] = kmeans_machine.get_variances_and_weights_for_each_cluster(training_set)
  >>> ubm.variances = variances
  >>> ubm.weights = weights

and a :py:class:`bob.learn.misc.ML_GMMTrainer` is used to compute the actual UBM model:

.. code-block:: python

  >>> trainer = bob.learn.misc.ML_GMMTrainer()
  >>> trainer.train(ubm, training_set)

After UBM training, the next step is the model enrollment.
Here, a separate GMM model is generated by shifting the UBM towards the mean of the model features [MM09]_.
For this purpose, we need to get the model images sorted by identity:

.. code-block:: python

  >>> model_ids = atnt_db.client_ids(groups = 'dev')

Now, we load the images for each identity, extract the DCT features and enroll a model for each identity.
For that purpose, a :py:class:`bob.learn.misc.MAP_GMMTrainer` is used:

.. code-block:: python

  >>> gmm_trainer = bob.learn.misc.MAP_GMMTrainer()
  >>> # ... initialize GMM trainer ...
  >>> for model_id in model_ids:
  ...   model_files = db.objects(groups = 'dev', purposes = 'enroll', client_ids = model_id)
  ...   model_feature_set_list = []
  ...   for model_file in model_files:
  ...     # ... load model image ...
  ...     model_dct_blocks = dct_extractor(model_image)
  ...     model_feature_set_list.append(model_dct_blocks)
  ...   model_feature_set = numpy.vstack(model_feature_set_list)
  ...   model_gmm = bob.learn.misc.GMMMachine(ubm)
  ...   gmm_trainer.train(model_gmm, model_feature_set)


Also the probe image need some processing.
First, of course, the DCT features are extracted.
Afterward, the :py:class:`bob.learn.misc.GMMStats` statistics for each probe file are generated:

.. code-block:: python

  >>> probe_image_files = atnt_db.objects(groups = 'dev', purposes = 'probe', ...)
  >>> for probe_file in probe_image_files:
  ...   # ... load probe image ...
  ...   probe_dct_blocks = dct_extractor(probe_image_blocks)
  ...   probe_gmm_stats = bob.learn.misc.GMMStats()
  ...   gmm_stats.init()
  ...   ubm.acc_statistics(probe_dct_blocks, probe_gmm_stats)

Finally, the scores for the probe files are computed using the :py:func:`bob.learn.misc.linear_scoring` function:

.. code-block:: python

  >>> for model_gmm in models:
  ...  for probe_gmm_stats in probes:
  ...    score = bob.learn.misc.linear_scoring([model_gmm], ubm, [probe_gmm_stats])[0,0]

Again, the evaluation of the scores is identical to the previous examples.
The expected ROC curve is:

.. image:: dct_ubm.png
  :scale: 100 %

The expected result is: FAR 5% and FRR 5% at distance threshold 7640.95


.. [TP91]   Matthew Turk and Alex Pentland. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):71-86, 1991.
.. [WFKM97] \L. Wiskott, J.-M. Fellous, N. Krüger and C.v.d. Malsburg. Face recognition by elastic bunch graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19:775-779, 1997.
.. [GHW12]  Manuel Günther, Dennis Haufe, Rolf P. Würtz. Face recognition with disparity corrected Gabor phase differences. in preparation
.. [MM09]   Chris McCool and Sébastien Marcel. Parts-based face verification using local frequency bands. In proceedings of IEEE/IAPR international conference on biometrics. 2009.
.. [RQD00]  D.A. Reynolds, T.F. Quatieri, and R.B. Dunn. Speaker verification using adapted gaussian mixture models. Digital Signal Processing, 10(1-3):19–41, 2000.
