Match sketch (drawing) of a face photo with digital color photo

Question

I am going to match the face of the sketch (drawing) in a color photograph. therefore, for research, I want to find out what are the problems associated with sketching, in complexion. until I find out that

• pixel difference in pixels
• texture difference
• distance difference
• and color (negligible effect)

I want to know (from a technical point of view) what other problems exist and what OPEN CV and JAVA CV methods and algorithms exist to overcome these problems?

Here is an example of sketches and photographs that are known to correspond to them:

Answer 1

This problem is called multimodal face recognition. There was great interest in comparing high-quality mugshot (modality 1) with images of poor quality of observation (modality 2), the other - front images for profiles or images for sketches like OP. Partial least squares (PLS) and related factor analysis (TFA) were used for this purpose.

The key difficulty is the calculation of two linear projections from the image in modality 1 (and modality 2) to the space, where two close points mean that the individual is the same. This is a key technical step. Here are some articles on this approach:

• Abhishek Sharma, David W Jacobs: Synthesis bypass: PLS for Face Recognition with a pose, low resolution and sketch . CVPR 2011.
• SJD Prince, JH Elder J. Warrell, F.M. Felisberti, Associated Factor Analysis for Face Recognition through Great Differences in Poses , IEEE Patt. Anal. Mach Intell, 30 (6), 970-984, 2008. Elder is a specialist in this field and various articles on this topic .
• B. Klare, Z. Li, and AK Jain, Corresponding judicial sketches for mugshot photos , IEEE Pattern Analysis and Machine Intelligence, September 29, 2010.

As you know, this is an active area of ​​research / problem. In terms of using OpenCV to overcome difficulties, let me give you an analogy: you need to build to build a house (coinciding with thumbnails for photographs), and you ask how it will work with Stanley's hammer (OpenCV). Of course, this will probably help. But you will also need many other resources: wood, time / money, pipes, cable, etc.

Answer 2

I think that the old work of James Elder on the completeness of a boundary map (using reconstruction by solving the Laplace equation) is very appropriate. See Results at the end of this article: http://elderlab.yorku.ca/~elder/publications/journals/ElderIJCV99.pdf

Answer 3

You could give Eigenfaces a try, although I never tested them with sketches, I think they could be a good starting point for your research.

See the Wiki: http://en.wikipedia.org/wiki/Eigenface and the OpenCV Tutorial: http://docs.opencv.org/modules/contrib/doc/facerec/facerec_tutorial.html (including not only Eigenfaces!)

Answer 4

OpenCV can be used to extract the features and machine learning needed to complete this task. I think you can start with the documents in the answers above, start with some basic functions and a prototype classifier with OpenCV.

I think you can also detect and map function points on faces. If you use this approach, you will have to run point-point detectors yourself (training Viola-Jones in OpenCV with your own data is an option).