Kernel optimization in discriminant analysis

Abstract

Kernel mapping is one of the most used approaches to intrinsically derive nonlinear classifiers. The idea is to use a kernel function which maps the original nonlinearly separable problem to a space of intrinsically larger dimensionality where the classes are linearly separable. A major problem in the design of kernel methods is to find the kernel parameters that make the problem linear in the mapped representation. This paper derives the first criterion that specifically aims to find a kernel representation where the Bayes classifier becomes linear. We illustrate how this result can be successfully applied in several kernel discriminant analysis algorithms. Experimental results, using a large number of databases and classifiers, demonstrate the utility of the proposed approach. The paper also shows (theoretically and experimentally) that a kernel version of Subclass Discriminant Analysis yields the highest recognition rates.

Fig. 1

Here we show an example of two non-linearly separable class distributions, each consisting of 3 subclasses. (a) Classification boundary of LDA. (b) SDA’s solution. Note how this solution is piecewise linear (i.e., linear when separating subclasses, but nonlinear when classifying classes). (c) KDA’s solution.

Fig. 2

Three examples of the use of the homoscedastic criterion, Q₁. The examples are for two Normal distributions with equal covariance matrix up to scale and rotation. (a) The value of Q₁ decreases as the angle θ increases. The 2D rotation between the two distributions θ is in the x axis. The value of Q₁ is in the y axis. (b) When θ = 0°, the two distributions are homoscedastic, and Q₁ takes its maximum value of .5. Note how for distributions that are close to homoscedastic (i.e., θ ≈ 0°), the value of the criterion remains high. (c) When θ = 45°, the value has decreased about .4. (d) By θ = 90°, Q₁ ≈ 3.

Fig. 3

Here we show a two class classification problem with multi-modal class distributions. When σ = 1 both KDA (a) and KSDA (b) generate solutions that have small training error. (c) However, when the model complexity is small, σ = 3, KDA fails. (d) KSDA’s solution resolves this problem with piecewise smooth, nonlinear classifiers.