AFAICT, custom kernels for SVM are not supported directly in OpenCV. It seems that LIBSVM, which is the base library that OpenCV uses for this, does not provide a particularly simple way to define custom kernels. Thus, many of the wrappers that use LIBSVM also do not provide this. It seems that there are several, for example. scikit for python: an example of scikit SVM with a custom kernel
You can also look at a completely different library, such as SVMlight . It directly supports custom kernels. Also see this SO question . There are several SVM libraries in the answers, as well as brief reviews.
If you have good reason to stay in OpenCV, you can execute it using the kernel type CvSVM::LINEAR and apply your custom kernel to the data before training SVM. I am a bit unclear as to whether this direction will be fruitful, so I hope that someone who has more experience with SVM can listen and comment. If you can use a “pre-computed kernel” by choosing “linear” as your kernel, and then see this answer for more ideas on how to proceed.
You can also consider enabling LIBSVM and invoking it directly, without using OpenCV. See FAQ # 418 for LIBSVM , which briefly talks about how to make custom kernels:
Q: I would like to use my own kernel. Any example? In svm.cpp, there are two routines for evaluating the kernel: k_function () and kernel_function (). Which one should I change?
Example: "LIBSVM for string data" in LIBSVM Tools.
The reason we have two functions is as follows. For the core RBF exp (-g | xi - xj | ^ 2), if we first calculate xi - xj and then the square of the norm, 3n operations are performed. Thus, we consider exp (-g (| xi | ^ 2 - 2dot (xi, xj) + | xj | ^ 2)) and, calculating all | xi | ^ 2 at the beginning, the number of operations is reduced to 2n. This is for learning. We cannot do this for forecasting, so a regular routine using these 3n operations is needed. The easiest way to have your own kernel is to put the same code in these two routines, replacing any core.
However, the latter option sounds a little painful. I would recommend scikit or SVMlight. Good luck to you!