tutorialCorr.dvi

1997), and (on artiﬁcial data) on the PET operator inversion problem (Vapnik, Golowich
and Smola, 1996). In most of these cases, SVM generalization performance (i.e. error rates
on test sets) either matches or is signiﬁcantly better than that of competing methods. The
use of SVMs for density estimation (Weston et al., 1997) and ANOVA decomposition (Stit-
son et al., 1997) has also been studied. Regarding extensions, the basic SVMs contain no
prior knowledge of the problem (for example, a large class of SVMs for the image recogni-
tion problem would give the same results if the pixels were ﬁrst permuted randomly (with
each image suﬀering the same permutation), an act of vandalism that would leave the best
performing neural networks severely handicapped) and much work has been done on in-
corporating prior knowledge into SVMs (Sch¨

olkopf, Burges and Vapnik, 1996; Sch¨

olkopf et

al., 1998a; Burges, 1998). Although SVMs have good generalization performance, they can
be abysmally slow in test phase, a problem addressed in (Burges, 1996; Osuna and Girosi,
1998). Recent work has generalized the basic ideas (Smola, Sch¨

olkopf and M¨

uller, 1998a;

Smola and Sch¨

olkopf, 1998), shown connections to regularization theory (Smola, Sch¨

olkopf

and M¨

uller, 1998b; Girosi, 1998; Wahba, 1998), and shown how SVM ideas can be incorpo-

rated in a wide range of other algorithms (Sch¨

olkopf, Smola and M¨

uller, 1998b; Sch¨

olkopf

et al, 1998c). The reader may also ﬁnd the thesis of (Sch¨

olkopf, 1997) helpful.

The problem which drove the initial development of SVMs occurs in several guises - the

bias variance tradeoﬀ (Geman, Bienenstock and Doursat, 1992), capacity control (Guyon
et al., 1992), overﬁtting (Montgomery and Peck, 1992) - but the basic idea is the same.
Roughly speaking, for a given learning task, with a given ﬁnite amount of training data, the
best generalization performance will be achieved if the right balance is struck between the
accuracy attained on that particular training set, and the “capacity” of the machine, that is,
the ability of the machine to learn any training set without error. A machine with too much
capacity is like a botanist with a photographic memory who, when presented with a new
tree, concludes that it is not a tree because it has a diﬀerent number of leaves from anything
she has seen before; a machine with too little capacity is like the botanist’s lazy brother,
who declares that if it’s green, it’s a tree. Neither can generalize well. The exploration and
formalization of these concepts has resulted in one of the shining peaks of the theory of
statistical learning (Vapnik, 1979).

In the following, bold typeface will indicate vector or matrix quantities; normal typeface

will be used for vector and matrix components and for scalars. We will label components
of vectors and matrices with Greek indices, and label vectors and matrices themselves with
Roman indices. Familiarity with the use of Lagrange multipliers to solve problems with
equality or inequality constraints is assumed

A Bound on the Generalization Performance of a Pattern Recognition Learn-
ing Machine

There is a remarkable family of bounds governing the relation between the capacity of a
learning machine and its performance

. The theory grew out of considerations of under what

circumstances, and how quickly, the mean of some empirical quantity converges uniformly,
as the number of data points increases, to the true mean (that which would be calculated
from an inﬁnite amount of data) (Vapnik, 1979). Let us start with one of these bounds.

The notation here will largely follow that of (Vapnik, 1995). Suppose we are given

observations. Each observation consists of a pair: a vector

∈

= 1

, . . . , l

and the

associated “truth”

, given to us by a trusted source. In the tree recognition problem,

might be a vector of pixel values (e.g.

= 256 for a 16x16 image), and

would be 1 if the

image contains a tree, and -1 otherwise (we use -1 here rather than 0 to simplify subsequent

formulae).

Now it is assumed that there exists some unknown probability distribution

(

, y

) from which these data are drawn, i.e., the data are assumed “iid” (independently

drawn and identically distributed). (We will use

for cumulative probability distributions,

and

for their densities). Note that this assumption is more general than associating a

ﬁxed

with every

: it allows there to be a distribution of

for a given

. In that case,

the trusted source would assign labels

according to a ﬁxed distribution, conditional on

. However, after this Section, we will be assuming ﬁxed

for given

Now suppose we have a machine whose task it is to learn the mapping

→

. The

machine is actually deﬁned by a set of possible mappings

→

(

, α

), where the functions

(

, α

) themselves are labeled by the adjustable parameters

. The machine is assumed to

be deterministic: for a given input

, and choice of

, it will always give the same output

(

, α

). A particular choice of

generates what we will call a “trained machine.” Thus,

for example, a neural network with ﬁxed architecture, with

corresponding to the weights

and biases, is a learning machine in this sense.

The expectation of the test error for a trained machine is therefore:

(

) =

−

(

, α

)

(

, y

)

(1)

Note that, when a density

(

, y

) exists,

(

, y

) may be written

(

, y

)

. This is a

nice way of writing the true mean error, but unless we have an estimate of what

(

, y

) is,

it is not very useful.

The quantity

(

) is called the expected risk, or just the risk. Here we will call it the

actual risk, to emphasize that it is the quantity that we are ultimately interested in. The
“empirical risk”

emp

(

) is deﬁned to be just the measured mean error rate on the training

set (for a ﬁxed, ﬁnite number of observations)

emp

(

) =

−

(

, α

)

(2)

Note that no probability distribution appears here.

emp

(

) is a ﬁxed number for a

particular choice of

and for a particular training set

{

, y

}

The quantity

−

(

, α

)

is called the loss. For the case described here, it can only

take the values 0 and 1. Now choose some

such that 0

≤

1. Then for losses taking

these values, with probability 1

−

, the following bound holds (Vapnik, 1995):

(

)

≤

emp

(

) +

(log(2

l/h

) + 1)

−

log(

η/

(3)

where

is a non-negative integer called the Vapnik Chervonenkis (VC) dimension, and is

a measure of the notion of capacity mentioned above. In the following we will call the right
hand side of Eq. (3) the “risk bound.” We depart here from some previous nomenclature:
the authors of (Guyon et al., 1992) call it the “guaranteed risk”, but this is something of a
misnomer, since it is really a bound on a risk, not a risk, and it holds only with a certain
probability, and so is not guaranteed. The second term on the right hand side is called the
“VC conﬁdence.”

We note three key points about this bound. First, remarkably, it is independent of

(

, y

It assumes only that both the training data and the test data are drawn independently
according to

some

(

, y

). Second, it is usually not possible to compute the left hand

side. Third, if we know

, we can easily compute the right hand side. Thus given several

diﬀerent learning machines (recall that “learning machine” is just another name for a family
of functions

(

, α

)), and choosing a ﬁxed, suﬃciently small

, by then taking that machine

which minimizes the right hand side, we are choosing that machine which gives the lowest
upper bound on the actual risk. This gives a principled method for choosing a learning
machine for a given task, and is the essential idea of structural risk minimization (see
Section 2.6). Given a ﬁxed family of learning machines to choose from, to the extent that
the bound is tight for at least one of the machines, one will not be able to do better than
this. To the extent that the bound is not tight for any, the hope is that the right hand
side still gives useful information as to which learning machine minimizes the actual risk.
The bound not being tight for the whole chosen family of learning machines gives critics a
justiﬁable target at which to ﬁre their complaints. At present, for this case, we must rely
on experiment to be the judge.

2.1.

The VC Dimension

The VC dimension is a property of a set of functions

{

(

)

}

(again, we use

as a generic set

of parameters: a choice of

speciﬁes a particular function), and can be deﬁned for various

classes of function

. Here we will only consider functions that correspond to the two-class

pattern recognition case, so that

(

, α

)

∈ {−

} ∀

, α

. Now if a given set of

points can

be labeled in all possible 2

ways, and for each labeling, a member of the set

{

(

)

}

can

be found which correctly assigns those labels, we say that that set of points is

shattered

that set of functions. The VC dimension for the set of functions

{

(

)

}

is deﬁned as the

maximum number of training points that can be shattered by

{

(

)

}

. Note that, if the VC

dimension is

, then there exists at least one set of

points that can be shattered, but it in

general it will not be true that

every

set of

points can be shattered.

2.2.

Shattering Points with Oriented Hyperplanes in

Suppose that the space in which the data live is

, and the set

{

(

)

}

consists of oriented

straight lines, so that for a given line, all points on one side are assigned the class 1, and all
points on the other side, the class

−

1. The orientation is shown in Figure 1 by an arrow,

specifying on which side of the line points are to be assigned the label 1. While it is possible
to ﬁnd three points that can be shattered by this set of functions, it is not possible to ﬁnd
four. Thus the VC dimension of the set of oriented lines in

is three.

Let’s now consider hyperplanes in

. The following theorem will prove useful (the proof

is in the Appendix):

Theorem 1

Consider some set of

points in

. Choose any one of the points as origin.

Then the

points can be shattered by oriented hyperplanes

if and only if the position

vectors of the remaining points are linearly independent

Corollary

: The VC dimension of the set of oriented hyperplanes in

+ 1, since we

can always choose

+ 1 points, and then choose one of the points as origin, such that the

position vectors of the remaining

points are linearly independent, but can never choose

+ 2 such points (since no

+ 1 vectors in

can be linearly independent).

An alternative proof of the corollary can be found in (Anthony and Biggs, 1995), and

references therein.

Figure 1.

Three points in

, shattered by oriented lines.

2.3.

The VC Dimension and the Number of Parameters

The VC dimension thus gives concreteness to the notion of the capacity of a given set
of functions. Intuitively, one might be led to expect that learning machines with many
parameters would have high VC dimension, while learning machines with few parameters
would have low VC dimension. There is a striking counterexample to this, due to E. Levin
and J.S. Denker (Vapnik, 1995): A learning machine with just one parameter, but with
inﬁnite VC dimension (a family of classiﬁers is said to have inﬁnite VC dimension if it can
shatter

points, no matter how large

). Deﬁne the step function

(

)

, x

∈

{

(

) =

∀

x >

(

) =

−

∀

≤

}

. Consider the one-parameter family of functions, deﬁned by

(

x, α

)

≡

(sin(

αx

))

, x, α

∈

(4)

You choose some number

, and present me with the task of ﬁnding

points that can be

shattered. I choose them to be:

= 10

−

, i

= 1

· · ·

, l.

(5)

You specify any labels you like:

, y

· · ·

, y

∈ {−

}

(6)

Then

(

) gives this labeling if I choose

to be

(1 +

−

)10

)

(7)

Thus the VC dimension of this machine is inﬁnite.

Interestingly, even though we can shatter an arbitrarily large number of points, we can

also ﬁnd just four points that cannot be shattered. They simply have to be equally spaced,
and assigned labels as shown in Figure 2. This can be seen as follows: Write the phase at

= 2

nπ

. Then the choice of label

= 1 requires 0

< δ < π

. The phase at

mod 2

, is 2

; then

= 1

⇒

< δ < π/

2. Similarly, point

forces

δ > π/

3. Then at

π/

< δ < π/

2 implies that

(

, α

) =

−

1, contrary to the assigned label. These four

points are the analogy, for the set of functions in Eq. (4), of the set of three points lying
along a line, for oriented hyperplanes in

. Neither set can be shattered by the chosen

family of functions.

with inﬁnite VC dimension, the bound is not even valid

. However, even though the bound

is not valid, nearest neighbour classiﬁers can still perform well. Thus this ﬁrst example is a
cautionary tale: inﬁnite “capacity” does not guarantee poor performance.

Let’s follow the time honoured tradition of understanding things by trying to break them,

and see if we can come up with a classiﬁer for which the bound is supposed to hold, but
which violates the bound. We want the left hand side of Eq. (3) to be as large as possible,
and the right hand side to be as small as possible. So we want a family of classiﬁers which
gives the worst possible actual risk of 0

5, zero empirical risk up to some number of training

observations, and whose VC dimension is easy to compute and is less than

(so that the

bound is non trivial). An example is the following, which I call the “notebook classiﬁer.”
This classiﬁer consists of a notebook with enough room to write down the classes of

training observations, where

≤

. For all subsequent patterns, the classiﬁer simply says

that all patterns have the same class. Suppose also that the data have as many positive
(

= +1) as negative (

−

1) examples, and that the samples are chosen randomly. The

notebook classiﬁer will have zero empirical risk for up to

observations; 0

5 training error

for all subsequent observations; 0

5 actual error, and VC dimension

. Substituting

these values in Eq. (3), the bound becomes:

≤

ln(2

l/m

) + 1

−

) ln(

η/

(8)

which is certainly met for all

(

) =

exp

(

−

≤

, z

≡

(

m/l

)

≤

(9)

which is true, since

(

) is monotonic increasing, and

(

= 1) = 0

236.

2.6.

Structural Risk Minimization

We can now summarize the principle of structural risk minimization (SRM) (Vapnik, 1979).
Note that the VC conﬁdence term in Eq. (3) depends on the chosen class of functions,
whereas the empirical risk and actual risk depend on the one particular function chosen by
the training procedure. We would like to ﬁnd that subset of the chosen set of functions, such
that the risk bound for that subset is minimized. Clearly we cannot arrange things so that
the VC dimension

varies smoothly, since it is an integer. Instead, introduce a “structure”

by dividing the entire class of functions into nested subsets (Figure 4). For each subset,
we must be able either to compute

, or to get a bound on

itself. SRM then consists of

ﬁnding that subset of functions which minimizes the bound on the actual risk. This can be
done by simply training a series of machines, one for each subset, where for a given subset
the goal of training is simply to minimize the empirical risk. One then takes that trained
machine in the series whose sum of empirical risk and VC conﬁdence is minimal.

We have now laid the groundwork necessary to begin our exploration of support vector

machines.

Linear Support Vector Machines

3.1.

The Separable Case

We will start with the simplest case: linear machines trained on separable data (as we shall
see, the analysis for the general case - nonlinear machines trained on non-separable data -

h1 < h2 < h3 ...

Figure 4.

Nested subsets of functions, ordered by VC dimension.

results in a very similar quadratic programming problem). Again label the training data

{

, y

}

= 1

· · ·

, l, y

∈ {−

}

∈

. Suppose we have some hyperplane which

separates the positive from the negative examples (a “separating hyperplane”). The points

which lie on the hyperplane satisfy

= 0, where

is normal to the hyperplane,

is the perpendicular distance from the hyperplane to the origin, and

is the

Euclidean norm of

. Let

(

−

) be the shortest distance from the separating hyperplane

to the closest positive (negative) example. Deﬁne the “margin” of a separating hyperplane
to be

−

. For the linearly separable case, the support vector algorithm simply looks for

the separating hyperplane with largest margin. This can be formulated as follows: suppose
that all the training data satisfy the following constraints:

≥

for

= +1

(10)

≤ −

for

−

(11)

These can be combined into one set of inequalities:

(

)

−

≥

∀

(12)

Now consider the points for which the equality in Eq. (10) holds (requiring that there

exists such a point is equivalent to choosing a scale for

and

). These points lie on

the hyperplane

= 1 with normal

and perpendicular distance from the

origin

−

. Similarly, the points for which the equality in Eq. (11) holds lie on the

hyperplane

−

1, with normal again

, and perpendicular distance from

the origin

| −

−

. Hence

−

= 1

and the margin is simply 2

. Note

that

and

are parallel (they have the same normal) and that no training points fall

between them. Thus we can ﬁnd the pair of hyperplanes which gives the maximum margin
by minimizing

, subject to constraints (12).

Thus we expect the solution for a typical two dimensional case to have the form shown

in Figure 5. Those training points for which the equality in Eq. (12) holds (i.e. those
which wind up lying on one of the hyperplanes

), and whose removal would change

the solution found, are called support vectors; they are indicated in Figure 5 by the extra
circles.

We will now switch to a Lagrangian formulation of the problem. There are two reasons

for doing this. The ﬁrst is that the constraints (12) will be replaced by constraints on the
Lagrange multipliers themselves, which will be much easier to handle. The second is that in
this reformulation of the problem, the training data will only appear (in the actual training
and test algorithms) in the form of dot products between vectors. This is a crucial property
which will allow us to generalize the procedure to the nonlinear case (Section 4).

-b
|w|

Origin

Margin

Figure 5.

Linear separating hyperplanes for the separable case. The support vectors are circled.

Thus, we introduce positive Lagrange multipliers

, i

= 1

· · ·

, l

, one for each of the

inequality constraints (12). Recall that the rule is that for constraints of the form

≥

the constraint equations are multiplied by

positive

Lagrange multipliers and subtracted

from the objective function, to form the Lagrangian. For equality constraints, the Lagrange
multipliers are unconstrained. This gives Lagrangian:

≡

−

(

) +

(13)

We must now minimize

with respect to

, b

, and simultaneously require that the

derivatives of

with respect to all the

vanish, all subject to the constraints

≥

(let’s call this particular set of constraints

). Now this is a convex quadratic programming

problem, since the objective function is itself convex, and those points which satisfy the
constraints also form a convex set (any linear constraint deﬁnes a convex set, and a set of

simultaneous linear constraints deﬁnes the intersection of

convex sets, which is also

a convex set). This means that we can equivalently solve the following “dual” problem:

maximize

, subject to the constraints that the gradient of

with respect to

and

vanish, and subject also to the constraints that the

≥

0 (let’s call

that

particular set of

constraints

). This particular dual formulation of the problem is called the Wolfe dual

(Fletcher, 1987). It has the property that the maximum of

, subject to constraints

occurs at the same values of the

and

, as the minimum of

, subject to constraints

Requiring that the gradient of

with respect to

and

vanish give the conditions:

(14)

= 0

(15)

Since these are equality constraints in the dual formulation, we can substitute them into

Eq. (13) to give

−

i,j

(16)

Note that we have now given the Lagrangian diﬀerent labels (

for primal,

for dual) to

emphasize that the two formulations are diﬀerent:

and

arise from the same objective

function but with diﬀerent constraints; and the solution is found by minimizing

or by

maximizing

. Note also that if we formulate the problem with

= 0, which amounts to

requiring that all hyperplanes contain the origin, the constraint (15) does not appear. This
is a mild restriction for high dimensional spaces, since it amounts to reducing the number
of degrees of freedom by one.

Support vector training (for the separable, linear case) therefore amounts to maximizing

with respect to the

, subject to constraints (15) and positivity of the

, with solution

given by (14). Notice that there is a Lagrange multiplier

for every training point. In

the solution, those points for which

0 are called “support vectors”, and lie on one of

the hyperplanes

, H

. All other training points have

= 0 and lie either on

(such that the equality in Eq. (12) holds), or on that side of

such that the

strict inequality in Eq. (12) holds. For these machines, the support vectors are the critical
elements of the training set. They lie closest to the decision boundary; if all other training
points were removed (or moved around, but so as not to cross

), and training was

repeated, the same separating hyperplane would be found.

3.2.

The Karush-Kuhn-Tucker Conditions

The Karush-Kuhn-Tucker (KKT) conditions play a central role in both the theory and
practice of constrained optimization. For the primal problem above, the KKT conditions
may be stated (Fletcher, 1987):

∂

∂w

−

iν

= 0

= 1

· · ·

, d

(17)

∂

∂b

−

= 0

(18)

(

)

−

≥

= 1

· · ·

, l

(19)

≥

∀

(20)

(

)

−

1) = 0

∀

(21)

The KKT conditions are satisﬁed at the solution of any constrained optimization problem

(convex or not), with any kind of constraints, provided that the intersection of the set
of feasible directions with the set of descent directions coincides with the intersection of
the set of feasible directions

for linearized constraints

with the set of descent directions

(see Fletcher, 1987; McCormick, 1983)). This rather technical regularity assumption holds
for all support vector machines, since the constraints are always linear. Furthermore, the
problem for SVMs is convex (a convex objective function, with constraints which give a
convex feasible region), and for convex problems (if the regularity condition holds), the
KKT conditions are

necessary and suﬃcient

for

, b, α

to be a solution (Fletcher, 1987).

Thus solving the SVM problem is equivalent to ﬁnding a solution to the KKT conditions.
This fact results in several approaches to ﬁnding the solution (for example, the primal-dual
path following method mentioned in Section 5).

As an immediate application, note that, while

is explicitly determined by the training

procedure, the threshold

is not, although it is implicitly determined. However

is easily

found by using the KKT “complementarity” condition, Eq. (21), by choosing any

for

which

= 0 and computing

(note that it is numerically safer to take the mean value of

resulting from all such equations).

Notice that all we’ve done so far is to cast the problem into an optimization problem

where the constraints are rather more manageable than those in Eqs. (10), (11). Finding
the solution for real world problems will usually require numerical methods. We will have
more to say on this later. However, let’s ﬁrst work out a rare case where the problem is
nontrivial (the number of dimensions is arbitrary, and the solution certainly not obvious),
but where the solution can be found analytically.

3.3.

Optimal Hyperplanes: An Example

While the main aim of this Section is to explore a non-trivial pattern recognition problem
where the support vector solution can be found analytically, the results derived here will
also be useful in a later proof. For the problem considered, every training point will turn
out to be a support vector, which is one reason we can ﬁnd the solution analytically.

Consider

+ 1 symmetrically placed points lying on a sphere

−

of radius

: more

precisely, the points form the vertices of an

-dimensional symmetric simplex. It is conve-

nient to embed the points in

in such a way that they all lie in the hyperplane which

passes through the origin and which is perpendicular to the (

+ 1)-vector (1

, ...,

1) (in this

formulation, the points lie on

−

, they span

, and are embedded in

). Explicitly,

recalling that vectors themselves are labeled by Roman indices and their coordinates by
Greek, the coordinates are given by:

iµ

−

i,µ

)

(

+ 1)

i,µ

+ 1

(22)

where the Kronecker delta,

i,µ

, is deﬁned by

i,µ

= 1 if

, 0 otherwise. Thus, for

example, the vectors for three equidistant points on the unit circle (see Figure 12) are:

= (

−

√

−

√

)

= (

−

√

−

√

)

= (

−

√

−

√

)

(23)

One consequence of the symmetry is that the angle between any pair of vectors is the

same (and is equal to arccos(

−

)):

(24)

−

(25)

or, more succinctly,

i,j

−

i,j

)

(26)

Assigning a class label

∈ {

−

}

arbitrarily to each point, we wish to ﬁnd that

hyperplane which separates the two classes with widest margin. Thus we must maximize

in Eq. (16), subject to

≥

0 and also subject to the equality constraint, Eq. (15).

Our strategy is to simply solve the problem as though there were no inequality constraints.
If the resulting solution does in fact satisfy

≥

∀

, then we will have found the general

solution, since the actual maximum of

will then lie in the feasible region, provided the

equality constraint, Eq. (15), is also met. In order to impose the equality constraint we
introduce an additional Lagrange multiplier

. Thus we seek to maximize

≡

−

i,j

−

(27)

where we have introduced the Hessian

≡

(28)

Setting

∂L

∂α

= 0 gives

(

)

λy

= 1

∀

(29)

Now

has a very simple structure: the oﬀ-diagonal elements are

−

, and the

diagonal elements are

. The fact that all the oﬀ-diagonal elements diﬀer only by factors

suggests looking for a solution which has the form:

1 +

−

(30)

where

and

are unknowns. Plugging this form in Eq. (29) gives:

+ 1

−

λy

(31)

where

is deﬁned by

≡

(32)

Thus

(

+ 1)

(33)

and substituting this into the equality constraint Eq. (15) to ﬁnd

gives

(

+ 1)

−

+ 1

, b

(

+ 1)

1 +

+ 1

(34)

which gives for the solution

(

+ 1)

−

+ 1

(35)

Also,

(

)

= 1

−

+ 1

(36)

Hence

i,j

−

+ 1

−

+ 1

(37)

Note that this is one of those cases where the Lagrange multiplier

can remain undeter-

mined (although determining it is trivial). We have now solved the problem, since all the

are clearly positive or zero (in fact the

will only be zero if all training points have

the same class). Note that

depends only on the

number

of positive (negative) polarity

points, and not on how the class labels are assigned to the points in Eq. (22). This is clearly
not true of

itself, which is given by

(

+ 1)

−

+ 1

(38)

The margin,

= 2

, is thus given by

−

(

+ 1))

)

(39)

Thus when the number of points

+ 1 is even, the minimum margin occurs when

= 0

(equal numbers of positive and negative examples), in which case the margin is

min

√

. If

+ 1 is odd, the minimum margin occurs when

1, in which case

min

(

+ 1)

(

√

+ 2). In both cases, the maximum margin is given by

max

(

+ 1)

Thus, for example, for the two dimensional simplex consisting of three points lying on

(and spanning

), and with labeling such that not all three points have the same polarity,

the maximum and minimum margin are both 3

2 (see Figure (12)).

Note that the results of this Section amount to an alternative, constructive proof that the

VC dimension of oriented separating hyperplanes in

is at least

+ 1.

3.4.

Test Phase

Once we have trained a Support Vector Machine, how can we use it? We simply determine
on which side of the decision boundary (that hyperplane lying half way between

and

and parallel to them) a given test pattern

lies and assign the corresponding class label,

i.e. we take the class of

to be

sgn

(

3.5.

The Non-Separable Case

The above algorithm for separable data, when applied to non-separable data, will ﬁnd no
feasible solution: this will be evidenced by the objective function (i.e. the dual Lagrangian)
growing arbitrarily large. So how can we extend these ideas to handle non-separable data?
We would like to relax the constraints (10) and (11), but only when necessary, that is, we
would like to introduce a further cost (i.e. an increase in the primal objective function) for
doing so. This can be done by introducing positive slack variables

, i

= 1

· · ·

, l

in the

constraints (Cortes and Vapnik, 1995), which then become:

≥

−

for

= +1

(40)

≤ −

1 +

for

−

(41)

≥

∀

(42)

Thus, for an error to occur, the corresponding

must exceed unity, so

is an upper

bound on the number of training errors. Hence a natural way to assign an extra cost
for errors is to change the objective function to be minimized from

2 to

2 +

(

)

, where

is a parameter to be chosen by the user, a larger

corresponding to

assigning a higher penalty to errors. As it stands, this is a convex programming problem
for any positive integer

; for

= 2 and

= 1 it is also a quadratic programming problem,

and the choice

= 1 has the further advantage that neither the

, nor their Lagrange

multipliers, appear in the Wolfe dual problem, which becomes:

Maximize:

≡

−

i,j

(43)

subject to:

≤

(44)

= 0

(45)

The solution is again given by

(46)

where

is the number of support vectors. Thus the only diﬀerence from the optimal

hyperplane case is that the

now have an upper bound of

. The situation is summarized

schematically in Figure 6.

We will need the Karush-Kuhn-Tucker conditions for the primal problem. The primal

Lagrangian is

−

{

(

)

−

1 +

} −

(47)

where the

are the Lagrange multipliers introduced to enforce positivity of the

. The

KKT conditions for the primal problem are therefore (note

runs from 1 to the number of

training points, and

from 1 to the dimension of the data)

∂L

∂w

−

iν

= 0

(48)

∂L

∂b

−

= 0

(49)

∂L

∂ξ

−

= 0

(50)

(

)

−

1 +

≥

(51)

≥

(52)

≥

(53)

≥

(54)

{

(

)

−

1 +

}

= 0

(55)

= 0

(56)

As before, we can use the KKT complementarity conditions, Eqs.

(55) and (56), to

determine the threshold

. Note that Eq. (50) combined with Eq. (56) shows that

= 0 if

< C

. Thus we can simply take any training point for which 0

< α

< C

to use Eq. (55)

(with

= 0) to compute

. (As before, it is numerically wiser to take the average over all

such training points.)

-b

−ξ

|w|

Figure 6.

Linear separating hyperplanes for the non-separable case.

3.6.

A Mechanical Analogy

Consider the case in which the data are in

. Suppose that the i’th support vector exerts

a force

on a stiﬀ sheet lying along the decision surface (the “decision sheet”)

(here ˆ

denotes the unit vector in the direction

). Then the solution (46) satisﬁes the

conditions of mechanical equilibrium:

Forces =

= 0

(57)

Torques =

∧

(

) = ˆ

∧

= 0

(58)

(Here the

are the support vectors, and

∧

denotes the vector product.) For data in

clearly the condition that the sum of forces vanish is still met. One can easily show that
the torque also vanishes.

This mechanical analogy depends only on the form of the solution (46), and therefore holds

for both the separable and the non-separable cases. In fact this analogy holds in general

(i.e., also for the nonlinear case described below). The analogy emphasizes the interesting
point that the “most important” data points are the support vectors with highest values of

, since they exert the highest forces on the decision sheet. For the non-separable case, the

upper bound

≤

corresponds to an upper bound on the force any given point is allowed

to exert on the sheet. This analogy also provides a reason (as good as any other) to call
these particular vectors “support vectors”

3.7.

Examples by Pictures

Figure 7 shows two examples of a two-class pattern recognition problem, one separable and
one not. The two classes are denoted by circles and disks respectively. Support vectors are
identiﬁed with an extra circle. The error in the non-separable case is identiﬁed with a cross.
The reader is invited to use Lucent’s SVM Applet (Burges, Knirsch and Haratsch, 1996) to
experiment and create pictures like these (if possible, try using 16 or 24 bit color).

Figure 7.

The linear case, separable (left) and not (right). The background colour shows the shape of the

decision surface.

Nonlinear Support Vector Machines

How can the above methods be generalized to the case where the decision function

is not

a linear function of the data? (Boser, Guyon and Vapnik, 1992), showed that a rather old
trick (Aizerman, 1964) can be used to accomplish this in an astonishingly straightforward
way. First notice that the only way in which the data appears in the training problem, Eqs.
(43) - (45), is in the form of dot products,

. Now suppose we ﬁrst mapped the data

to some other (possibly inﬁnite dimensional) Euclidean space

, using a mapping which we

will call Φ:

Φ :

→ H

(59)

Then of course the training algorithm would only depend on the data through dot products

, i.e. on functions of the form Φ(

)

Φ(

). Now if there were a “kernel function”

such that

(

) = Φ(

)

Φ(

), we would only need to use

in the training algorithm,

and would never need to explicitly even know what Φ is. One example is

(

) =

−

(60)

In this particular example,

is inﬁnite dimensional, so it would not be very easy to work

with Φ explicitly. However, if one replaces

(

) everywhere in the training

algorithm, the algorithm will happily produce a support vector machine which lives in an
inﬁnite dimensional space, and furthermore do so in roughly the same amount of time it
would take to train on the un-mapped data. All the considerations of the previous sections
hold, since we are still doing a linear separation, but in a diﬀerent space.

But how can we use this machine? After all, we need

, and that will live in

also (see

Eq. (46)). But in test phase an SVM is used by computing dot products of a given test
point

with

, or more speciﬁcally by computing the sign of

(

) =

Φ(

)

Φ(

) +

(

) +

(61)

where the

are the support vectors. So again we can avoid computing Φ(

) explicitly

and use the

(

) = Φ(

)

Φ(

) instead.

Let us call the space in which the data live,

. (Here and below we use

as a mnemonic

for “low dimensional”, and

for “high dimensional”: it is usually the case that the range of

Φ is of much higher dimension than its domain). Note that, in addition to the fact that

lives in

, there will in general be no vector in

which maps, via the map Φ, to

. If there

were,

(

) in Eq. (61) could be computed in one step, avoiding the sum (and making the

corresponding SVM

times faster, where

is the number of support vectors). Despite

this, ideas along these lines can be used to signiﬁcantly speed up the test phase of SVMs
(Burges, 1996). Note also that it is easy to ﬁnd kernels (for example, kernels which are
functions of the dot products of the

) such that the training algorithm and solution

found are independent of the dimension of both

and

In the next Section we will discuss which functions

are allowable and which are not.

Let us end this Section with a very simple example of an allowed kernel, for which we

can

construct the mapping Φ.

Suppose that your data are vectors in

, and you choose

(

) = (

)

. Then

it’s easy to ﬁnd a space

, and mapping Φ from

, such that (

)

= Φ(

)

Φ(

we choose

and

Φ(

) =

⎛
⎝

√

⎞
⎠

(62)

(note that here the subscripts refer to vector components). For data in

deﬁned on the

square [

−

[

−

∈

(a typical situation, for grey level image data), the (entire)

image of Φ is shown in Figure 8. This Figure also illustrates how to think of this mapping:
the image of Φ may live in a space of very high dimension, but it is just a (possibly very
contorted) surface whose intrinsic dimension

is just that of

Note that neither the mapping Φ nor the space

are unique for a given kernel. We could

equally well have chosen

to again be

and

Φ(

) =

√

⎛
⎝

(

−

)

(

)

⎞
⎠

(63)

0.2

0.4

0.6

0.8

-1

-0.5

0.5

0.2

0.4

0.6

0.8

Figure 8.

Image, in

, of the square [

−

[

−

∈

under the mapping Φ.

to be

and

Φ(

) =

⎛
⎜

⎜

⎝

⎞
⎟

⎟

⎠

(64)

The literature on SVMs usually refers to the space

as a Hilbert space, so let’s end this

Section with a few notes on this point. You can think of a Hilbert space as a generalization
of Euclidean space that behaves in a gentlemanly fashion. Speciﬁcally, it is any linear space,
with an inner product deﬁned, which is also complete with respect to the corresponding
norm (that is, any Cauchy sequence of points converges to a point in the space). Some
authors (e.g. (Kolmogorov, 1970)) also require that it be separable (that is, it must have a
countable subset whose closure is the space itself), and some (e.g. Halmos, 1967) don’t. It’s a
generalization mainly because its inner product can be

any

inner product, not just the scalar

(“dot”) product used here (and in Euclidean spaces in general). It’s interesting that the
older mathematical literature (e.g. Kolmogorov, 1970) also required that Hilbert spaces be
inﬁnite dimensional, and that mathematicians are quite happy deﬁning inﬁnite dimensional
Euclidean spaces. Research on Hilbert spaces centers on operators in those spaces, since
the basic properties have long since been worked out. Since some people understandably
blanch at the mention of Hilbert spaces, I decided to use the term Euclidean throughout
this tutorial.

4.1.

Mercer’s Condition

For which kernels does there exist a pair

}

, with the properties described above, and

for which does there not? The answer is given by Mercer’s condition (Vapnik, 1995; Courant
and Hilbert, 1953): There exists a mapping Φ and an expansion

(

) =

Φ(

)

Φ(

)

(65)

if and only if, for any

(

) such that

(

)

is ﬁnite

(66)

then

(

)

(

)

(

)

≥

(67)

Note that for speciﬁc cases, it may not be easy to check whether Mercer’s condition is

satisﬁed. Eq. (67) must hold for

every

with ﬁnite

norm (i.e. which satisﬁes Eq. (66)).

However, we can easily prove that the condition is satisﬁed for positive integral powers of
the dot product:

(

) = (

)

. We must show that

(

)

(

)

(

)

≥

(68)

The typical term in the multinomial expansion of (

d
i

)

contributes a term of the

form

· · ·

(

−

· · ·

(

)

(

)

(69)

to the left hand side of Eq. (67), which factorizes:

· · ·

(

−

· · ·

(

· · ·

(

)

≥

(70)

One simple consequence is that any kernel which can be expressed as

(

) =

∞

(

)

, where the

are positive real coeﬃcients and the series is uniformly convergent, satisﬁes

Mercer’s condition, a fact also noted in (Smola, Sch¨

olkopf and M¨

uller, 1998b).

Finally, what happens if one uses a kernel which does not satisfy Mercer’s condition? In

general, there may exist data such that the Hessian is indeﬁnite, and for which the quadratic
programming problem will have no solution (the dual objective function can become arbi-
trarily large). However, even for kernels that do not satisfy Mercer’s condition, one might
still ﬁnd that a given training set results in a positive semideﬁnite Hessian, in which case the
training will converge perfectly well. In this case, however, the geometrical interpretation
described above is lacking.

4.2.

Some Notes on

and

Mercer’s condition tells us whether or not a prospective kernel is actually a dot product
in some space, but it does not tell us how to construct Φ or even what

is. However, as

with the homogeneous (that is, homogeneous in the dot product in

) quadratic polynomial

kernel discussed above, we can explicitly construct the mapping for some kernels. In Section
6.1 we show how Eq. (62) can be extended to arbitrary homogeneous polynomial kernels,
and that the corresponding space

is a Euclidean space of dimension

−

. Thus for

example, for a degree

= 4 polynomial, and for data consisting of 16 by 16 images (d=256),

dim(

) is 183,181,376.

Usually, mapping your data to a “feature space” with an enormous number of dimensions

would bode ill for the generalization performance of the resulting machine. After all, the

set of all hyperplanes

{

, b

}

are parameterized by dim(

) +1 numbers. Most pattern

recognition systems with billions, or even an inﬁnite, number of parameters would not make
it past the start gate. How come SVMs do so well? One might argue that, given the form
of solution, there are at most

+ 1 adjustable parameters (where

is the number of training

samples), but this seems to be begging the question

. It must be something to do with our

requirement of

maximum margin

hyperplanes that is saving the day. As we shall see below,

a strong case can be made for this claim.

Since the mapped surface is of intrinsic dimension dim(

), unless dim(

) = dim(

), it

is obvious that the mapping cannot be onto (surjective). It also need not be one to one
(bijective): consider

→ −

, x

→ −

in Eq. (62). The image of Φ need not itself be

a vector space: again, considering the above simple quadratic example, the vector

−

Φ(

) is

not in the image of Φ unless

= 0. Further, a little playing with the inhomogeneous kernel

(

) = (

+ 1)

(71)

will convince you that the corresponding Φ can map two vectors that are linearly dependent

onto two vectors that are linearly independent in

So far we have considered cases where Φ is done implicitly. One can equally well turn

things around and

start

with Φ, and then construct the corresponding kernel. For example

(Vapnik, 1996), if

, then a Fourier expansion in the data

, cut oﬀ after

terms,

has the form

(

) =

(

cos(

) +

sin(

))

(72)

and this can be viewed as a dot product between two vectors in

= (

√

, a

, . . . , a

, . . .

and the mapped Φ(

) = (

√

cos(

)

cos(2

)

, . . . ,

sin(

)

sin(2

)

, . . .

). Then the correspond-

ing (Dirichlet) kernel can be computed in closed form:

Φ(

)

Φ(

) =

(

, x

) =

sin((

+ 1

2)(

−

))

2 sin((

−

)

(73)

This is easily seen as follows: letting

≡

−

Φ(

)

Φ(

) =

cos(

) cos(

) + sin(

) sin(

)

−

cos(

rδ

) =

−

{

(

irδ

)

}

−

{

−

(

+1)

)

−

iδ

)

}

= (sin((

+ 1

))

2 sin(

δ/

Finally, it is clear that the above implicit mapping trick will work for

any

algorithm in

which the data only appear as dot products (for example, the nearest neighbor algorithm).
This fact has been used to derive a nonlinear version of principal component analysis by
(Sch¨

olkopf, Smola and M¨

uller, 1998b); it seems likely that this trick will continue to ﬁnd

uses elsewhere.

4.3.

Some Examples of Nonlinear SVMs

The ﬁrst kernels investigated for the pattern recognition problem were the following:

(

) = (

+ 1)

(74)

(

) =

−

(75)

(

) = tanh(

−

)

(76)

Eq. (74) results in a classiﬁer that is a polynomial of degree

in the data; Eq. (75) gives

a Gaussian radial basis function classiﬁer, and Eq. (76) gives a particular kind of two-layer
sigmoidal neural network. For the RBF case, the number of centers (

in Eq. (61)),

the centers themselves (the

), the weights (

), and the threshold (

) are all produced

automatically

by the SVM training and give excellent results compared to classical RBFs,

for the case of Gaussian RBFs (Sch¨

olkopf et al, 1997). For the neural network case, the

ﬁrst layer consists of

sets of weights, each set consisting of

(the dimension of the

data) weights, and the second layer consists of

weights (the

), so that an evaluation

simply requires taking a weighted sum of sigmoids, themselves evaluated on dot products of
the test data with the support vectors. Thus for the neural network case, the architecture
(number of weights) is determined by SVM training.

Note, however, that the hyperbolic tangent kernel only satisﬁes Mercer’s condition for

certain values of the parameters

and

(and of the data

). This was ﬁrst noticed

experimentally (Vapnik, 1995); however some necessary conditions on these parameters for
positivity are now known

Figure 9 shows results for the same pattern recognition problem as that shown in Figure

7, but where the kernel was chosen to be a cubic polynomial. Notice that, even though
the number of degrees of freedom is higher, for the linearly separable case (left panel), the
solution is roughly linear, indicating that the capacity is being controlled; and that the
linearly non-separable case (right panel) has become separable.

Figure 9.

Degree 3 polynomial kernel. The background colour shows the shape of the decision surface.

Finally, note that although the SVM classiﬁers described above are binary classiﬁers, they

are easily combined to handle the multiclass case. A simple, eﬀective combination trains

one-versus-rest classiﬁers (say, “one” positive, “rest” negative) for the

-class case and

takes the class for a test point to be that corresponding to the largest positive distance
(Boser, Guyon and Vapnik, 1992).

4.4.

Global Solutions and Uniqueness

When is the solution to the support vector training problem global, and when is it unique?
By “global”, we mean that there exists no other point in the feasible region at which the
objective function takes a lower value. We will address two kinds of ways in which uniqueness
may not hold: solutions for which

{

, b

}

are themselves unique, but for which the expansion

in Eq. (46) is not; and solutions whose

{

, b

}

diﬀer. Both are of interest: even if the

pair

{

, b

}

is unique, if the

are not, there may be equivalent expansions of

which

require fewer support vectors (a trivial example of this is given below), and which therefore
require fewer instructions during test phase.

It turns out that every local solution is also global. This is a property of any convex

programming problem (Fletcher, 1987).

Furthermore, the solution is guaranteed to be

unique if the objective function (Eq. (43)) is strictly convex, which in our case means
that the Hessian must be positive deﬁnite (note that for quadratic objective functions

the Hessian is positive deﬁnite if and only if

is strictly convex; this is not true for non-

quadratic

: there, a positive deﬁnite Hessian implies a strictly convex objective function,

but not vice versa (consider

) (Fletcher, 1987)). However, even if the Hessian is

positive semideﬁnite, the solution can still be unique: consider two points along the real
line with coordinates

= 1 and

= 2, and with polarities + and

−

. Here the Hessian is

positive semideﬁnite, but the solution (

−

, b

= 3

, ξ

= 0 in Eqs. (40), (41), (42)) is

unique. It is also easy to ﬁnd solutions which are not unique in the sense that the

in the

expansion of

are not unique:: for example, consider the problem of four separable points

on a square in

= [1

1],

= [

−

1],

= [

−

1] and

= [1

−

1], with polarities

−

+] respectively. One solution is

= [1

0],

= 0,

= [0

25];

another has the same

and

, but

= [0

0] (note that both solutions satisfy the

constraints

0 and

= 0). When can this occur in general? Given some solution

, choose an

which is in the null space of the Hessian

, and require that

be orthogonal to the vector all of whose components are 1. Then adding

in Eq.

(43) will leave

unchanged. If 0

≤

and

satisﬁes Eq. (45), then

also a solution

How about solutions where the

{

, b

}

are themselves not unique? (We emphasize that

this can only happen in principle if the Hessian is not positive deﬁnite, and even then,
the solutions are necessarily global). The following very simple theorem shows that if non-
unique solutions occur, then the solution at one optimal point is continuously deformable
into the solution at the other optimal point, in such a way that all intermediate points are
also solutions.

Theorem 2

Let the variable

stand for the pair of variables

{

, b

}

. Let the Hessian for

the problem be positive semideﬁnite, so that the objective function is convex. Let

and

be two points at which the objective function attains its minimal value. Then there exists a
path

(

) = (1

−

)

, τ

∈

, such that

(

)

is a solution for all

Proof:

Let the minimum value of the objective function be

min

. Then by assumption,

(

) =

(

) =

min

. By convexity of

(

))

≤

−

)

(

) +

τ F

(

) =

min

Furthermore, by linearity, the

(

) satisfy the constraints Eq. (40), (41): explicitly (again

combining both constraints into one):

(

) =

((1

−

)(

) +

(

))

≥

−

)(1

−

) +

−

) = 1

−

(77)

Although simple, this theorem is quite instructive. For example, one might think that the

problems depicted in Figure 10 have several diﬀerent optimal solutions (for the case of linear
support vector machines). However, since one cannot smoothly move the hyperplane from
one proposed solution to another without generating hyperplanes which are not solutions,
we know that these proposed solutions are in fact not solutions at all. In fact, for each
of these cases, the optimal unique solution is at

= 0, with a suitable choice of

(which

has the eﬀect of assigning the same label to all the points). Note that this is a perfectly
acceptable solution to the classiﬁcation problem: any proposed hyperplane (with

= 0)

will cause the primal objective function to take a higher value.

Figure 10.

Two problems, with proposed (incorrect) non-unique solutions.

Finally, note that the fact that SVM training always ﬁnds a global solution is in contrast

to the case of neural networks, where many local minima usually exist.

Methods of Solution

The support vector optimization problem can be solved analytically only when the number
of training data is very small, or for the separable case when it is known beforehand which
of the training data become support vectors (as in Sections 3.3 and 6.2). Note that this can
happen when the problem has some symmetry (Section 3.3), but that it can also happen
when it does not (Section 6.2). For the general analytic case, the worst case computational
complexity is of order

(inversion of the Hessian), where

is the number of support

vectors, although the two examples given both have complexity of O(1).

However, in most real world cases, Equations (43) (with dot products replaced by ker-

nels), (44), and (45) must be solved numerically. For small problems, any general purpose
optimization package that solves linearly constrained convex quadratic programs will do. A
good survey of the available solvers, and where to get them, can be found

in (Mor´

e and

Wright, 1993).

For larger problems, a range of existing techniques can be brought to bear. A full ex-

ploration of the relative merits of these methods would ﬁll another tutorial. Here we just
describe the general issues, and for concreteness, give a brief explanation of the technique
we currently use. Below, a “face” means a set of points lying on the boundary of the feasible
region, and an “active constraint” is a constraint for which the equality holds. For more

on nonlinear programming techniques see (Fletcher, 1987; Mangasarian, 1969; McCormick,
1983).

The basic recipe is to (1) note the optimality (KKT) conditions which the solution must

satisfy, (2) deﬁne a strategy for approaching optimality by uniformly increasing the dual
objective function subject to the constraints, and (3) decide on a decomposition algorithm
so that only portions of the training data need be handled at a given time (Boser, Guyon
and Vapnik, 1992; Osuna, Freund and Girosi, 1997a). We give a brief description of some
of the issues involved. One can view the problem as requiring the solution of a sequence
of equality constrained problems. A given equality constrained problem can be solved in
one step by using the Newton method (although this requires storage for a factorization of
the projected Hessian), or in at most

steps using conjugate gradient ascent (Press et al.,

1992) (where

is the number of data points for the problem currently being solved: no extra

storage is required). Some algorithms move within a given face until a new constraint is
encountered, in which case the algorithm is restarted with the new constraint added to the
list of equality constraints. This method has the disadvantage that only one new constraint
is made active at a time. “Projection methods” have also been considered (Mor´

e, 1991),

where a point outside the feasible region is computed, and then line searches and projections
are done so that the actual move remains inside the feasible region. This approach can add
several new constraints at once. Note that in both approaches, several active constraints
can become

inactive

in one step. In all algorithms, only the essential part of the Hessian

(the columns corresponding to

= 0) need be computed (although some algorithms do

compute the whole Hessian). For the Newton approach, one can also take advantage of the
fact that the Hessian is positive semideﬁnite by diagonalizing it with the Bunch-Kaufman
algorithm (Bunch and Kaufman, 1977; Bunch and Kaufman, 1980) (if the Hessian were
indeﬁnite, it could still be easily reduced to 2x2 block diagonal form with this algorithm).
In this algorithm, when a new constraint is made active or inactive, the factorization of
the projected Hessian is easily updated (as opposed to recomputing the factorization from
scratch). Finally, in interior point methods, the variables are essentially rescaled so as to
always remain inside the feasible region. An example is the “LOQO” algorithm of (Vander-
bei, 1994a; Vanderbei, 1994b), which is a primal-dual path following algorithm. This last
method is likely to be useful for problems where the number of support vectors as a fraction
of training sample size is expected to be large.

We brieﬂy describe the core optimization method we currently use

. It is an active set

method combining gradient and conjugate gradient ascent. Whenever the objective function
is computed, so is the gradient, at very little extra cost. In phase 1, the search directions

are along the gradient. The nearest face along the search direction is found. If the dot

product of the gradient there with

indicates that the maximum along

lies between the

current point and the nearest face, the optimal point along the search direction is computed
analytically (note that this does not require a line search), and phase 2 is entered. Otherwise,
we jump to the new face and repeat phase 1. In phase 2, Polak-Ribiere conjugate gradient
ascent (Press et al., 1992) is done, until a new face is encountered (in which case phase 1 is
re-entered) or the stopping criterion is met. Note the following:

•

Search directions are always projected so that the

continue to satisfy the equality

constraint Eq. (45). Note that the conjugate gradient algorithm will still work; we
are simply searching in a subspace. However, it is important that this projection is
implemented in such a way that not only is Eq. (45) met (easy), but also so that the
angle between the resulting search direction, and the search direction prior to projection,
is minimized (not quite so easy).

•

We also use a “sticky faces” algorithm: whenever a given face is hit more than once,
the search directions are adjusted so that all subsequent searches are done within that
face. All “sticky faces” are reset (made “non-sticky”) when the rate of increase of the
objective function falls below a threshold.

•

The algorithm stops when the fractional rate of increase of the objective function

falls

below a tolerance (typically 1e-10, for double precision). Note that one can also use
as stopping criterion the condition that the size of the projected search direction falls
below a threshold. However, this criterion does not handle scaling well.

•

In my opinion the hardest thing to get right is handling precision problems correctly
everywhere. If this is not done, the algorithm may not converge, or may be much slower
than it needs to be.

A good way to check that your algorithm is working is to check that the solution satisﬁes

all the Karush-Kuhn-Tucker conditions for the primal problem, since these are necessary
and suﬃcient conditions that the solution be optimal. The KKT conditions are Eqs. (48)
through (56), with dot products between data vectors replaced by kernels wherever they
appear (note

must be expanded as in Eq. (48) ﬁrst, since

is not in general the mapping

of a point in

). Thus to check the KKT conditions, it is suﬃcient to check that the

satisfy 0

≤

, that the equality constraint (49) holds, that all points for which

≤

< C

satisfy Eq. (51) with

= 0, and that all points with

satisfy Eq. (51)

for some

≥

0. These are suﬃcient conditions for all the KKT conditions to hold: note

that by doing this we never have to explicitly compute the

, although doing so is

trivial.

5.1.

Complexity, Scalability, and Parallelizability

Support vector machines have the following very striking property. Both training and test
functions depend on the data only through the kernel functions

(

). Even though it

corresponds to a dot product in a space of dimension

, where

can be very large or

inﬁnite, the complexity of computing

can be far smaller. For example, for kernels of the

form

= (

)

, a dot product in

would require of order

−

operations, whereas

the computation of

(

) requires only

(

) operations (recall

is the dimension of

the data). It is this fact that allows us to construct hyperplanes in these very high dimen-
sional spaces yet still be left with a tractable computation. Thus SVMs circumvent both
forms of the “curse of dimensionality”: the proliferation of parameters causing intractable
complexity, and the proliferation of parameters causing overﬁtting.

5.1.1.

Training

For concreteness, we will give results for the computational complexity of

one the the above training algorithms (Bunch-Kaufman)

(Kaufman, 1998). These results

assume that diﬀerent strategies are used in diﬀerent situations. We consider the problem of
training on just one “chunk” (see below). Again let

be the number of training points,

the number of support vectors (SVs), and

the dimension of the input data. In the case

where most SVs are not at the upper bound, and

/l <<

1, the number of operations

(

+ (

)

). If instead

≈

1, then

(

) (basically

by starting in the interior of the feasible region). For the case where most SVs are at the
upper bound, and

/l <<

1, then

(

). Finally, if most SVs are at the

upper bound, and

≈

1, we have

(

For larger problems, two decomposition algorithms have been proposed to date. In the

“chunking” method (Boser, Guyon and Vapnik, 1992), one starts with a small, arbitrary

subset of the data and trains on that. The rest of the training data is tested on the resulting
classiﬁer, and a list of the errors is constructed, sorted by how far on the wrong side of the
margin they lie (i.e. how egregiously the KKT conditions are violated). The next chunk is
constructed from the ﬁrst

of these, combined with the

support vectors already found,

where

is decided heuristically (a chunk size that is allowed to grow too quickly or

too slowly will result in slow overall convergence). Note that vectors can be dropped from
a chunk, and that support vectors in one chunk may not appear in the ﬁnal solution. This
process is continued until all data points are found to satisfy the KKT conditions.

The above method requires that the number of support vectors

be small enough so that

a Hessian of size

will ﬁt in memory. An alternative decomposition algorithm has

been proposed which overcomes this limitation (Osuna, Freund and Girosi, 1997b). Again,
in this algorithm, only a small portion of the training data is trained on at a given time, and
furthermore, only a subset of the support vectors need be in the “working set” (i.e. that set
of points whose

’s are allowed to vary). This method has been shown to be able to easily

handle a problem with 110,000 training points and 100,000 support vectors. However, it
must be noted that the speed of this approach relies on many of the support vectors having
corresponding Lagrange multipliers

at the upper bound,

These training algorithms may take advantage of parallel processing in several ways. First,

all elements of the Hessian itself can be computed simultaneously. Second, each element
often requires the computation of dot products of training data, which could also be par-
allelized. Third, the computation of the objective function, or gradient, which is a speed
bottleneck, can be parallelized (it requires a matrix multiplication). Finally, one can envision
parallelizing at a higher level, for example by training on diﬀerent chunks simultaneously.
Schemes such as these, combined with the decomposition algorithm of (Osuna, Freund and
Girosi, 1997b), will be needed to make very large problems (i.e.

100,000 support vectors,

with many not at bound), tractable.

5.1.2.

Testing

In test phase, one must simply evaluate Eq.

(61), which will require

(

M N

) operations, where

is the number of operations required to evaluate the kernel.

For dot product and RBF kernels,

(

), the dimension of the data vectors. Again,

both the evaluation of the kernel and of the sum are highly parallelizable procedures.

In the absence of parallel hardware, one can still speed up test phase by a large factor, as

described in Section 9.

The VC Dimension of Support Vector Machines

We now show that the VC dimension of SVMs can be very large (even inﬁnite). We will
then explore several arguments as to why, in spite of this, SVMs usually exhibit good
generalization performance. However it should be emphasized that these are essentially
plausibility arguments. Currently there exists no theory which

guarantees

that a given

family of SVMs will have high accuracy on a given problem.

We will call any kernel that satisﬁes Mercer’s condition a positive kernel, and the corre-

sponding space

the embedding space. We will also call any embedding space with minimal

dimension for a given kernel a “minimal embedding space”. We have the following

Theorem 3

Let

be a positive kernel which corresponds to a minimal embedding space

. Then the VC dimension of the corresponding support vector machine (where the error

penalty

in Eq. (44) is allowed to take all values) is

dim

(

) + 1

Proof:

If the minimal embedding space has dimension

, then

points in the image of

under the mapping Φ can be found whose position vectors in

are linearly independent.

From Theorem 1, these vectors can be shattered by hyperplanes in

. Thus by either

restricting ourselves to SVMs for the separable case (Section 3.1), or for which the error
penalty

is allowed to take all values (so that, if the points are linearly separable, a

can

be found such that the solution does indeed separate them), the family of support vector
machines with kernel

can also shatter these points, and hence has VC dimension

+ 1.

Let’s look at two examples.

6.1.

The VC Dimension for Polynomial Kernels

Consider an SVM with homogeneous polynomial kernel, acting on data in

(

) = (

)

∈

(78)

As in the case when

= 2 and the kernel is quadratic (Section 4), one can explicitly

construct the map Φ. Letting

, so that

(

) = (

· · ·

)

, we see that

each dimension of

corresponds to a term with given powers of the

in the expansion of

. In fact if we choose to label the components of Φ(

) in this manner, we can explicitly

write the mapping for any

and

···

(

) =

· · ·

p, r

≥

(79)

This leads to

Theorem 4

If the space in which the data live has dimension

(i.e.

), the

dimension of the minimal embedding space, for homogeneous polynomial kernels of degree

(

) = (

)

∈

), is

−

(The proof is in the Appendix). Thus the VC dimension of SVMs with these kernels is

−

+ 1. As noted above, this gets very large very quickly.

6.2.

The VC Dimension for Radial Basis Function Kernels

Theorem 5

Consider the class of Mercer kernels for which

(

)

→

−

→

∞

, and for which

(

)

is O(1), and assume that the data can be chosen arbitrarily from

. Then the family of classiﬁers consisting of support vector machines using these kernels,

and for which the error penalty is allowed to take all values, has inﬁnite VC dimension.

Proof:

The kernel matrix,

≡

(

), is a Gram matrix (a matrix of dot products:

see (Horn, 1985)) in

. Clearly we can choose training data such that all oﬀ-diagonal

elements

can be made arbitrarily small, and by assumption all diagonal elements

are of O(1). The matrix

is then of full rank; hence the set of vectors, whose dot products

form

, are linearly independent (Horn, 1985); hence, by Theorem 1, the points can be

shattered by hyperplanes in

, and hence also by support vector machines with suﬃciently

large error penalty. Since this is true for any ﬁnite number of points, the VC dimension of
these classiﬁers is inﬁnite.

Note that the assumptions in the theorem are stronger than necessary (they were chosen

to make the connection to radial basis functions clear). In fact it is only necessary that

training points can be chosen such that the rank of the matrix

increases without limit

increases. For example, for Gaussian RBF kernels, this can also be accomplished (even

for training data restricted to lie in a bounded subset of

) by choosing small enough

RBF widths. However in general the VC dimension of SVM RBF classiﬁers can certainly be
ﬁnite, and indeed, for data restricted to lie in a bounded subset of

, choosing restrictions

on the RBF widths is a good way to control the VC dimension.

This case gives us a second opportunity to present a situation where the SVM solution

can be computed analytically, which also amounts to a second, constructive proof of the
Theorem. For concreteness we will take the case for Gaussian RBF kernels of the form

(

) =

−

. Let us choose training points such that the smallest distance

between any pair of points is much larger than the width

. Consider the decision function

evaluated on the support vector

(

) =

−

(80)

The sum on the right hand side will then be largely dominated by the term

; in fact

the ratio of that term to the contribution from the rest of the sum can be made arbitrarily
large by choosing the training points to be arbitrarily far apart. In order to ﬁnd the SVM
solution, we again assume for the moment that every training point becomes a support
vector, and we work with SVMs for the separable case (Section 3.1) (the same argument
will hold for SVMs for the non-separable case if

in Eq. (44) is allowed to take large

enough values).

Since all points are support vectors, the equalities in Eqs. (10), (11)

will hold for them. Let there be

(

−

) positive (negative) polarity points. We further

assume that all positive (negative) polarity points have the same value

(

−

) for their

Lagrange multiplier. (We will know that this assumption is correct if it delivers a solution
which satisﬁes all the KKT conditions and constraints). Then Eqs. (19), applied to all the
training data, and the equality constraint Eq. (18), become

= 1

−

= 0

(81)

which are satisﬁed by

−

(82)

Thus, since the resulting

are also positive, all the KKT conditions and constraints are

satisﬁed, and we must have found the global solution (with zero training errors). Since the
number of training points, and their labeling, is arbitrary, and they are separated without
error, the VC dimension is inﬁnite.

The situation is summarized schematically in Figure 11.

Figure 11.

Gaussian RBF SVMs of suﬃciently small width can classify an arbitrarily large number of

training points correctly, and thus have inﬁnite VC dimension

Now we are left with a striking conundrum. Even though their VC dimension is inﬁnite (if

the data is allowed to take all values in

), SVM RBFs can have excellent performance

(Sch¨

olkopf et al, 1997). A similar story holds for polynomial SVMs. How come?

The Generalization Performance of SVMs

In this Section we collect various arguments and bounds relating to the generalization perfor-
mance of SVMs. We start by presenting a family of SVM-like classiﬁers for which structural
risk minimization can be rigorously implemented, and which will give us some insight as to
why maximizing the margin is so important.

7.1.

VC Dimension of Gap Tolerant Classiﬁers

Consider a family of classiﬁers (i.e. a set of functions Φ on

) which we will call “gap

tolerant classiﬁers.” A particular classiﬁer

∈

Φ is speciﬁed by the location and diameter

of a ball in

, and by two hyperplanes, with parallel normals, also in

. Call the set of

points lying between, but not on, the hyperplanes the “margin set.” The decision functions

are deﬁned as follows: points that lie inside the ball, but not in the margin set, are assigned

class

{±

}

, depending on which side of the margin set they fall. All other points are simply

deﬁned to be “correct”, that is, they are not assigned a class by the classiﬁer, and do not
contribute to any risk. The situation is summarized, for

= 2, in Figure 12. This rather

odd family of classiﬁers, together with a condition we will impose on how they are trained,
will result in systems very similar to SVMs, and for which structural risk minimization can
be demonstrated. A rigorous discussion is given in the Appendix.

Label the diameter of the ball

and the perpendicular distance between the two hyper-

planes

. The VC dimension is deﬁned as before to be the maximum number of points that

can be shattered by the family, but by “shattered” we mean that the points can occur as

errors

in all possible ways (see the Appendix for further discussion). Clearly we can control

the VC dimension of a family of these classiﬁers by controlling the minimum margin

and maximum diameter

that members of the family are allowed to assume. For example,

consider the family of gap tolerant classiﬁers in

with diameter

= 2, shown in Figure

12. Those with margin satisfying

≤

2 can shatter three points; if 3

< M <

2, they

can shatter two; and if

≥

2, they can shatter only one. Each of these three families of

classiﬁers corresponds to one of the sets of classiﬁers in Figure 4, with just three nested
subsets of functions, and with

= 1,

= 2, and

= 3.

M = 3/2

D = 2

Φ=0

Φ=1

Φ=−1

Φ=0

Figure 12.

A gap tolerant classiﬁer on data in

These ideas can be used to show how gap tolerant classiﬁers implement structural risk

minimization.

The extension of the above example to spaces of arbitrary dimension is

encapsulated in a (modiﬁed) theorem of (Vapnik, 1995):

Theorem 6

For data in

, the VC dimension

of gap tolerant classiﬁers of minimum

margin

min

and maximum diameter

max

is bounded above

by min

{

max

min

, d

}

For the proof we assume the following lemma, which in (Vapnik, 1979) is held to follow

from symmetry arguments

Lemma

: Consider

≤

+ 1 points lying in a ball

∈

. Let the points be shatterable

by gap tolerant classiﬁers with margin

. Then in order for

to be maximized, the points

must lie on the vertices of an (

−

1)-dimensional symmetric simplex, and must also lie on

the surface of the ball.

Proof:

We need only consider the case where the number of points

satisﬁes

≤

+ 1.

(

n > d

+ 1 points will not be shatterable, since the VC dimension of oriented hyperplanes in

+ 1, and any distribution of points which can be shattered by a gap tolerant classiﬁer

can also be shattered by an oriented hyperplane; this also shows that

≤

+ 1). Again we

consider points on a sphere of diameter

, where the sphere itself is of dimension

−

2. We

will need two results from Section 3.3, namely (1) if

is even, we can ﬁnd a distribution of

points (the vertices of the (

−

1)-dimensional symmetric simplex) which can be shattered by

gap tolerant classiﬁers if

max

min

−

1, and (2) if

is odd, we can ﬁnd a distribution

points which can be so shattered if

max

min

= (

−

(

+ 1)

is even, at most

points can be shattered whenever

−

≤

max

min

< n.

(83)

Thus for

even the maximum number of points that can be shattered may be written

max

min

+ 1.

is odd, at most

points can be shattered when

max

min

= (

−

(

+ 1)

However, the quantity on the right hand side satisﬁes

−

(

−

(

+ 1)

< n

−

(84)

for all integer

n >

1. Thus for

odd the largest number of points that can be shattered

is certainly bounded above by

max

min

+ 1, and from the above this bound is also

satisﬁed when

is even. Hence in general the VC dimension

of gap tolerant classiﬁers

must satisfy

≤

max

min

+ 1

(85)

This result, together with

≤

+ 1, concludes the proof.

7.2.

Gap Tolerant Classiﬁers, Structural Risk Minimization, and SVMs

Let’s see how we can do structural risk minimization with gap tolerant classiﬁers. We need
only consider that subset of the Φ, call it Φ

, for which training “succeeds”, where by success

we mean that all training data are assigned a label

∈ {±

}

(note that these labels do not

have to coincide with the actual labels, i.e. training errors are allowed). Within Φ

, ﬁnd

the subset which gives the fewest training errors - call this number of errors

min

. Within

that subset, ﬁnd the function

which gives maximum margin (and hence the lowest bound

on the VC dimension). Note the value of the resulting risk bound (the right hand side of
Eq. (3), using the bound on the VC dimension in place of the VC dimension). Next, within
Φ

, ﬁnd that subset which gives

min

+ 1 training errors. Again, within that subset, ﬁnd

the

which gives the maximum margin, and note the corresponding risk bound. Iterate,

and take that classiﬁer which gives the overall minimum risk bound.

An alternative approach is to divide the functions Φ into nested subsets Φ

, i

∈ Z

, i

≥

as follows: all

∈

have

{

D, M

}

satisfying

≤

. Thus the family of functions

in Φ

has VC dimension bounded above by min(

i, d

) + 1. Note also that Φ

⊂

. SRM

then proceeds by taking that

for which training succeeds in each subset and for which

the empirical risk is minimized in that subset, and again, choosing that

which gives the

lowest overall risk bound.

Note that it is essential to these arguments that the bound (3) holds for

any

chosen decision

function, not just the one that minimizes the empirical risk (otherwise eliminating solutions
for which some training point

satisﬁes

(

) = 0 would invalidate the argument).

The resulting gap tolerant classiﬁer is in fact a special kind of support vector machine

which simply does not count data falling outside the sphere containing all the training data,
or inside the separating margin, as an error. It seems very reasonable to conclude that
support vector machines, which are trained with very similar objectives, also gain a similar
kind of capacity control from their training. However, a gap tolerant classiﬁer is not an
SVM, and so the argument does not constitute a rigorous demonstration of structural risk
minimization for SVMs. The original argument for structural risk minimization for SVMs is
known to be ﬂawed, since the structure there is determined by the data (see (Vapnik, 1995),
Section 5.11). I believe that there is a further subtle problem with the original argument.
The structure is deﬁned so that no training points are members of the margin set. However,
one must still specify how test points that fall into the margin are to be labeled. If one simply

assigns the same, ﬁxed class to them (say +1), then the VC dimension will be higher

than

the bound derived in Theorem 6. However, the same is true if one labels them all as errors
(see the Appendix). If one labels them all as “correct”, one arrives at gap tolerant classiﬁers.

On the other hand, it is known how to do structural risk minimization for systems where

the structure does depend on the data (Shawe-Taylor et al., 1996a; Shawe-Taylor et al.,
1996b). Unfortunately the resulting bounds are much looser than the VC bounds above,
which are already very loose (we will examine a typical case below where the VC bound is
a factor of 100 higher than the measured test error). Thus at the moment structural risk
minimization alone does not provide a

rigorous

explanation as to why SVMs often have good

generalization performance. However, the above arguments strongly suggest that algorithms
that minimize

can be expected to give better generalization performance. Further

evidence for this is found in the following theorem of (Vapnik, 1998), which we quote without
proof

Theorem 7

For optimal hyperplanes passing through the origin, we have

[

(

error

)]

≤

[

]

(86)

where

(

error

)

is the probability of error on the test set, the expectation on the left is over

all training sets of size

−

, and the expectation on the right is over all training sets of size

However, in order for these observations to be useful for real problems, we need a way to

compute the diameter of the minimal enclosing sphere described above, for any number of
training points and for any kernel mapping.

7.3.

How to Compute the Minimal Enclosing Sphere

Again let Φ be the mapping to the embedding space

. We wish to compute the radius

of the smallest sphere in

which encloses the mapped training data: that is, we wish to

minimize

subject to

Φ(

)

−

≤

∀

(87)

where

∈ H

is the (unknown) center of the sphere. Thus introducing positive Lagrange

multipliers

, the primal Lagrangian is

−

(

−

Φ(

)

−

)

(88)

This is again a convex quadratic programming problem, so we can instead maximize the

Wolfe dual

(

)

−

i,j

(

)

(89)

(where we have again replaced Φ(

)

Φ(

) by

(

)) subject to:

= 1

(90)

≥

(91)

with solution given by

Φ(

)

(92)

Thus the problem is very similar to that of support vector training, and in fact the code

for the latter is easily modiﬁed to solve the above problem. Note that we were in a sense
“lucky”, because the above analysis shows us that there

exists

an expansion (92) for the

center; there is no

a priori

reason why we should expect that the center of the sphere in

should be expressible in terms of the mapped training data in this way. The same can be
said of the solution for the support vector problem, Eq. (46). (Had we chosen some other
geometrical construction, we might not have been so fortunate. Consider the smallest area
equilateral triangle containing two given points in

. If the points’ position vectors are

linearly dependent, the center of the triangle cannot be expressed in terms of them.)

7.4.

A Bound from Leave-One-Out

(Vapnik, 1995) gives an alternative bound on the actual risk of support vector machines:

[

(

error

)]

≤

[Number of support vectors]

Number of training samples

(93)

where

(

error

) is the actual risk for a machine trained on

−

1 examples,

[

(

error

)]

is the expectation of the actual risk over all choices of training set of size

−

1, and

[Number of support vectors] is the expectation of the number of support vectors over all

choices of training sets of size

. It’s easy to see how this bound arises: consider the typical

situation after training on a given training set, shown in Figure 13.

Figure 13.

Support vectors (circles) can become errors (cross) after removal and re-training (the dotted line

denotes the new decision surface).

We can get an estimate of the test error by removing one of the training points, re-training,

and then testing on the removed point; and then repeating this, for all training points. From
the support vector solution we know that removing any training points that are not support
vectors (the latter include the errors) will have no eﬀect on the hyperplane found. Thus
the worst that can happen is that every support vector will become an error. Taking the
expectation over all such training sets therefore gives an upper bound on the actual risk,
for training sets of size

−

Although elegant, I have yet to ﬁnd a use for this bound. There seem to be many situations

where the actual error increases even though the number of support vectors decreases, so
the intuitive conclusion (systems that give fewer support vectors give better performance)
often seems to fail. Furthermore, although the bound can be tighter than that found using
the estimate of the VC dimension combined with Eq. (3), it can at the same time be less
predictive, as we shall see in the next Section.

7.5.

VC, SV Bounds and the Actual Risk

Let us put these observations to some use. As mentioned above, training an SVM RBF
classiﬁer will automatically give values for the RBF weights, number of centers, center
positions, and threshold. For Gaussian RBFs, there is only one parameter left: the RBF
width (

in Eq. (80)) (we assume here only one RBF width for the problem). Can we

ﬁnd the optimal value for that too, by choosing that

which minimizes

? Figure 14

shows a series of experiments done on 28x28 NIST digit data, with 10,000 training points and
60,000 test points. The top curve in the left hand panel shows the VC bound (i.e. the bound
resulting from approximating the VC dimension in Eq. (3)

by Eq. (85)), the middle curve

shows the bound from leave-one-out (Eq. (93)), and the bottom curve shows the measured
test error. Clearly, in this case, the bounds are very loose. The right hand panel shows just
the VC bound (the top curve, for

200), together with the test error, with the latter

scaled up by a factor of 100 (note that the two curves cross). It is striking that the two
curves have minima in the same place: thus in this case, the VC bound, although loose,
seems to be nevertheless predictive. Experiments on digits 2 through 9 showed that the VC
bound gave a minimum for which

was within a factor of two of that which minimized the

test error (digit 1 was inconclusive). Interestingly, in those cases the VC bound consistently
gave a lower prediction for

than that which minimized the test error. On the other hand,

the leave-one-out bound, although tighter, does not seem to be predictive, since it had no
minimum for the values of

tested.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

100 200 300 400 500 600 700 800 900 1000

Actual Risk : SV Bound : VC Bound

Sigma Squared

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

100 200 300 400 500 600 700 800 900 1000

VC Bound : Actual Risk * 100

Sigma Squared

Figure 14.

The VC bound can be predictive even when loose.

Acknowledgments

I’m very grateful to P. Knirsch, C. Nohl, E. Osuna, E. Rietman, B. Sch¨

olkopf, Y. Singer, A.

Smola, C. Stenard, and V. Vapnik, for their comments on the manuscript. Thanks also to
the reviewers, and to the Editor, U. Fayyad, for extensive, useful comments. Special thanks
are due to V. Vapnik, under whose patient guidance I learned the ropes; to A. Smola and
B. Sch¨

olkopf, for many interesting and fruitful discussions; and to J. Shawe-Taylor and D.

Schuurmans, for valuable discussions on structural risk minimization.

Appendix

A.1.

Proofs of Theorems

We collect here the theorems stated in the text, together with their proofs. The Lemma has
a shorter proof using a “Theorem of the Alternative,” (Mangasarian, 1969) but we wished
to keep the proofs as self-contained as possible.

Lemma 1

Two sets of points in

may be separated by a hyperplane if and only if the

intersection of their convex hulls is empty.

Proof:

We allow the notions of points in

, and position vectors of those points, to be

used interchangeably in this proof. Let

be the convex hulls of two sets of points

. Let

−

denote the set of points whose position vectors are given by

−

∈

(note that

−

does not contain the origin), and let

−

have

the corresponding meaning for the convex hulls. Then showing that

and

are linearly

separable (separable by a hyperplane) is equivalent to showing that the set

−

is linearly

separable from the origin

. For suppose the latter: then

∃

∈

, b

∈

, b <

0 such

that

b >

∀

∈

−

. Now pick some

∈

, and denote the set of all points

−

∈

−

. Then

b >

∀

∈

−

, and clearly

b <

, so the sets

−

and

are linearly separable. Repeating this process

shows that

−

is linearly separable from the origin if and only if

and

are linearly

separable.

We now show that, if

∅

, then

−

is linearly separable from the origin.

Clearly

−

does not contain the origin.

Furthermore

−

is convex, since

∀

−

, λ

∈

, we have (1

−

)

((1

−

)

−

((1

−

)

∈

−

. Hence it is suﬃcient to show that any

convex set

, which does not contain

, is linearly separable from

. Let

min

∈

that point whose Euclidean distance from

min

, is minimal. (Note there can be only

one such point, since if there were two, the chord joining them, which also lies in

, would

contain points closer to

.) We will show that

∀

∈

min

0. Suppose

∃

∈

such that

min

≤

0. Let

be the line segment joining

min

and

. Then convexity

implies that

⊂

. Thus

O /

∈

, since by assumption

O /

∈

. Hence the three points

and

min

form an obtuse (or right) triangle, with obtuse (or right) angle occurring at the

point

. Deﬁne ˆ

≡

(

−

min

)

−

min

. Then the distance from the closest point in

min

−

(

min

)

, which is less than

min

. Hence

min

0 and

linearly separable from

. Thus

−

is linearly separable from

, and

a fortiori

−

is linearly separable from

, and thus

is linearly separable from

It remains to show that, if the two sets of points

are linearly separable, the intersec-

tion of their convex hulls if empty. By assumption there exists a pair

∈

, b

∈

, such

that

∀

∈

b >

0 and

∀

∈

b <

0. Consider a general point

∈

. It

may be written

= 1

≤

1. Then

{

}

Similarly, for points

∈

b <

0. Hence

∅

, since otherwise we

would be able to ﬁnd a point

which simultaneously satisﬁes both inequalities.

Theorem 1

: Consider some set of

points in

. Choose any one of the points as

origin. Then the

points can be shattered by oriented hyperplanes if and only if the

position vectors of the remaining points are linearly independent.

Proof:

Label the origin

, and assume that the

−

1 position vectors of the remaining

points are linearly independent. Consider any partition of the

points into two subsets,

and

, of order

and

respectively, so that

. Let

be the subset

containing

. Then the convex hull

is that set of points whose position vectors

satisfy

= 1

, α

≥

(A.1)

where the

are the position vectors of the

points in

(including the null position

vector of the origin). Similarly, the convex hull

is that set of points whose position

vectors

satisfy

= 1

, β

≥

(A.2)

where the

are the position vectors of the

points in

. Now suppose that

and

intersect. Then there exists an

∈

which simultaneously satisﬁes Eq. (A.1) and Eq.

(A.2). Subtracting these equations gives a linear combination of the

−

1 non-null position

vectors which vanishes, which contradicts the assumption of linear independence. By the
lemma, since

and

do not intersect, there exists a hyperplane separating

and

Since this is true for any choice of partition, the

points can be shattered.

It remains to show that if the

−

1 non-null position vectors are not linearly independent,

then the

points cannot be shattered by oriented hyperplanes. If the

−

1 position vectors

are not linearly independent, then there exist

−

1 numbers,

, such that

−

= 0

(A.3)

If all the

are of the same sign, then we can scale them so that

∈

1] and

= 1.

Eq. (A.3) then states that the origin lies in the convex hull of the remaining points; hence,
by the lemma, the origin cannot be separated from the remaining points by a hyperplane,
and the points cannot be shattered.

If the

are not all of the same sign, place all the terms with negative

on the right:

∈

(A.4)

where

are the indices of the corresponding partition of

(i.e. of the set

with the

origin removed). Now scale this equation so that either

∈

= 1 and

∈

| ≤

∈

| ≤

1 and

∈

= 1. Suppose without loss of generality that the latter

holds. Then the left hand side of Eq. (A.4) is the position vector of a point lying in the

convex hull of the points

{

∈

}

(or, if the equality holds, of the points

{

∈

}

and the right hand side is the position vector of a point lying in the convex hull of the points

∈

, so the convex hulls overlap, and by the lemma, the two sets of points cannot be

separated by a hyperplane. Thus the

points cannot be shattered.

Theorem 4

: If the data is

-dimensional (i.e.

), the dimension of the mini-

mal embedding space, for homogeneous polynomial kernels of degree

(

) = (

)

∈

), is

−

Proof:

First we show that the the number of components of Φ(

) is

−

. Label the

components of Φ as in Eq. (79). Then a component is uniquely identiﬁed by the choice
of the

integers

≥

d
i

. Now consider

objects distributed amongst

−

partitions (numbered 1 through

−

1), such that objects are allowed to be to the left of

all partitions, or to the right of all partitions. Suppose

objects fall between partitions

and

+ 1. Let this correspond to a term

in the product in Eq. (79). Similarly,

objects falling to the left of all partitions corresponds to a term

, and

objects falling

to the right of all partitions corresponds to a term

. Thus the number of distinct terms

of the form

· · ·

d
i

≥

0 is the number of way of distributing

the objects and partitions amongst themselves, modulo permutations of the partitions and
permutations of the objects, which is

−

Next we must show that the set of vectors with components Φ

···

(

) span the space

This follows from the fact that the components of Φ(

) are linearly independent functions.

For suppose instead that the image of Φ acting on

∈ L

is a subspace of

. Then there

exists a ﬁxed nonzero vector

∈ H

such that

dim

(

)

(

) = 0

∀

∈ L

(A.5)

Using the labeling introduced above, consider a particular component of Φ:

···

(

)

(A.6)

Since Eq. (A.5) holds for all

, and since the mapping Φ in Eq. (79) certainly has all

derivatives deﬁned, we can apply the operator

(

∂

∂x

)

· · ·

(

∂

∂x

)

(A.7)

to Eq. (A.5), which will pick that one term with corresponding powers of the

in Eq.

(79), giving

···

= 0

(A.8)

Since this is true for all choices of

· · ·

, r

such that

d
i

, every component of

must vanish. Hence the image of Φ acting on

∈ L

spans

A.2.

Gap Tolerant Classiﬁers and VC Bounds

The following point is central to the argument. One normally thinks of a collection of points
as being “shattered” by a set of functions, if for any choice of labels for the points, a function

from the set can be found which assigns those labels to the points. The VC dimension of that
set of functions is then deﬁned as the maximum number of points that can be so shattered.
However, consider a slightly diﬀerent deﬁnition. Let a set of points be shattered by a set
of functions if for any choice of labels for the points, a function from the set can be found
which assigns the

incorrect

labels to all the points. Again let the VC dimension of that set

of functions be deﬁned as the maximum number of points that can be so shattered.

It is in fact this second deﬁnition (which we adopt from here on) that enters the VC

bound proofs (Vapnik, 1979; Devroye, Gy¨

orﬁ and Lugosi, 1996). Of course for functions

whose range is

{±

}

(i.e. all data will be assigned either positive or negative class), the two

deﬁnitions are the same. However, if all points falling in some region are simply deemed to
be “errors”, or “correct”, the two deﬁnitions are diﬀerent. As a concrete example, suppose
we deﬁne “gap intolerant classiﬁers”, which are like gap tolerant classiﬁers, but which label
all points lying in the margin or outside the sphere as

errors

. Consider again the situation in

Figure 12, but assign positive class to all three points. Then a gap intolerant classiﬁer with
margin width greater than the ball diameter cannot shatter the points if we use the ﬁrst
deﬁnition of “shatter”, but can shatter the points if we use the second (correct) deﬁnition.

With this caveat in mind, we now outline how the VC bounds can apply to functions with

range

{±

}

, where the label 0 means that the point is labeled “correct.” (The bounds

will also apply to functions where 0 is deﬁned to mean “error”, but the corresponding VC
dimension will be higher, weakening the bound, and in our case, making it useless). We will
follow the notation of (Devroye, Gy¨

orﬁ and Lugosi, 1996).

Consider points

∈

, and let

(

) denote a density on

. Let

be a function on

with range

{±

}

, and let Φ be a set of such functions. Let each

have an associated

label

∈ {±

}

. Let

{

· · ·

}

be any ﬁnite number of points in

: then we require Φ

to have the property that there exists at least one

∈

Φ such that

(

)

∈ {±

} ∀

. For

given

, deﬁne the set of points

{

= 1

, φ

(

) =

−

} ∪ {

−

, φ

(

) = 1

}

(A.9)

We require that the

be such that all sets

are measurable. Let

denote the set of all

Deﬁnition

: Let

, i

= 1

· · ·

, n

points. We deﬁne the empirical risk for the set

{

, φ

}

to be

(

{

, φ

}

) = (1

)

∈

(A.10)

where

is the indicator function. Note that the empirical risk is zero if

(

) = 0

∀

Deﬁnition

: We deﬁne the actual risk for the function

to be

(

) =

(

∈

)

(A.11)

Note also that those points

for which

(

) = 0 do not contribute to the actual risk.

Deﬁnition

: For ﬁxed (

· · ·

)

∈

, let

be the number of diﬀerent sets in

{{

· · ·

} ∩

∈ A}

(A.12)

where the sets

are deﬁned above. The n-th shatter coeﬃcient of

is deﬁned

(

, n

) =

max

···

∈{

}

(

· · ·

)

(A.13)

We also deﬁne the VC dimension for the class

to be the maximum integer

≥

1 for

which

(

, k

) = 2

Theorem 8

(adapted from Devroye, Gy¨

orﬁ and Lugosi, 1996, Theorem 12.6):Given

(

{

, φ

}

)

(

)

and

(

, n

)

deﬁned above, and given

points

(

, ...,

)

∈

, let

denote that sub-

set of

such that all

∈

satisfy

(

)

∈ {±

} ∀

. (This restriction may be viewed as

part of the training algorithm). Then for any such

(

{

, φ

}

)

−

(

)

≤

(

, n

) exp

−

(A.14)

The proof is exactly that of (Devroye, Gy¨

orﬁ and Lugosi, 1996), Sections 12.3, 12.4 and

12.5, Theorems 12.5 and 12.6. We have dropped the “sup” to emphasize that this holds
for any of the functions

. In particular, it holds for those

which minimize the empirical

error and for which all training data take the values

{±

}

. Note however that the proof

only holds for the second deﬁnition of shattering given above. Finally, note that the usual
form of the VC bounds is easily derived from Eq. (A.14) by using

(

, n

)

≤

(

en/h

)

(where

is the VC dimension) (Vapnik, 1995), setting

= 8

(

, n

) exp

−

, and solving for

Clearly these results apply to our gap tolerant classiﬁers of Section 7.1. For them, a

particular classiﬁer

∈

Φ is speciﬁed by a set of parameters

{

B, H, M

}

, where

is a

ball in

∈

is the diameter of

is a

−

1 dimensional oriented hyperplane in

, and

∈

is a scalar which we have called the margin.

itself is speciﬁed by its

normal (whose direction speciﬁes which points

(

−

) are labeled positive (negative) by

the function), and by the minimal distance from

to the origin. For a given

∈

Φ, the

margin set

is deﬁned as the set consisting of those points whose minimal distance to

is less than

2. Deﬁne

≡

, and

−

≡

−

. The function

is then deﬁned as follows:

(

) = 1

∀

∈

, φ

(

) =

−

∀

∈

−

, φ

(

) = 0 otherwise

(A.15)

and the corresponding sets

as in Eq. (A.9).

Notes

1. K. M¨

uller, Private Communication

2. The reader in whom this elicits a sinking feeling is urged to study (Strang, 1986; Fletcher,

1987; Bishop, 1995). There is a simple geometrical interpretation of Lagrange multipliers: at
a boundary corresponding to a single constraint, the gradient of the function being extremized
must be parallel to the gradient of the function whose contours specify the boundary. At a
boundary corresponding to the intersection of constraints, the gradient must be parallel to a
linear combination (non-negative in the case of inequality constraints) of the gradients of the
functions whose contours specify the boundary.

3. In this paper, the phrase “learning machine” will be used for any function estimation algo-

rithm, “training” for the parameter estimation procedure, “testing” for the computation of the
function value, and “performance” for the generalization accuracy (i.e. error rate as test set
size tends to inﬁnity), unless otherwise stated.

4. Given the name “test set,” perhaps we should also use “train set;” but the hobbyists got there

ﬁrst.

5. We use the term “oriented hyperplane” to emphasize that the mathematical object considered

is the pair

{

}

, where

is the set of points which lie in the hyperplane and

is a particular

choice for the unit normal. Thus

{

}

and

{

−

}

are diﬀerent oriented hyperplanes.

6. Such a set of

points (which span an

−

1 dimensional subspace of a linear space) are said

to be “in general position” (Kolmogorov, 1970). The convex hull of a set of

points in general

position deﬁnes an

−

1 dimensional simplex, the vertices of which are the points themselves.

7. The derivation of the bound assumes that the empirical risk converges uniformly to the actual

risk as the number of training observations increases (Vapnik, 1979). A necessary and suﬃcient
condition for this is that lim

→∞

(

)

= 0, where

is the number of training samples and

(

) is the VC entropy of the set of decision functions (Vapnik, 1979; Vapnik, 1995). For any

set of functions with inﬁnite VC dimension, the VC entropy is

log 2: hence for these classiﬁers,

the required uniform convergence does not hold, and so neither does the bound.

8. There is a nice geometric interpretation for the dual problem: it is basically ﬁnding the two

closest points of convex hulls of the two sets. See (Bennett and Bredensteiner, 1998).

9. One can deﬁne the torque to be

...µ

−

...µ

−

(A.16)

where repeated indices are summed over on the right hand side, and where

is the totally

antisymmetric tensor with

...n

= 1. (Recall that Greek indices are used to denote tensor

components). The sum of torques on the decision sheet is then:

...µ

iµ

−

iµ

...µ

iµ

−

...µ

−

= 0

(A.17)

10. In the original formulation (Vapnik, 1979) they were called “extreme vectors.”

11. By “decision function” we mean a function

(

) whose sign represents the class assigned to

data point

12. By “intrinsic dimension” we mean the number of parameters required to specify a point on the

manifold.

13. Alternatively one can argue that, given the form of the solution, the possible

must lie in a

subspace of dimension

14. Work in preparation.

15. Thanks to A. Smola for pointing this out.

16. Many thanks to one of the reviewers for pointing this out.

17. The core quadratic optimizer is about 700 lines of C++. The higher level code (to handle

caching of dot products, chunking, IO, etc) is quite complex and considerably larger.

18. Thanks to L. Kaufman for providing me with these results.

19. Recall that the “ceiling” sign

means “smallest integer greater than or equal to.” Also, there

is a typo in the actual formula given in (Vapnik, 1995), which I have corrected here.

20. Note, for example, that the distance between every pair of vertices of the symmetric simplex

is the same: see Eq. (26). However, a rigorous proof is needed, and as far as I know is lacking.

21. Thanks to J. Shawe-Taylor for pointing this out.

22. V. Vapnik, Private Communication.

23. There is an alternative bound one might use, namely that corresponding to the set of totally

bounded non-negative functions (Equation (3.28) in (Vapnik, 1995)). However, for loss func-
tions taking the value zero or one, and if the empirical risk is zero, this bound is looser than

that in Eq. (3) whenever

(log(2

l/h

)+1)

−

log(

η/

16, which is the case here.

24. V. Blanz, Private Communication

References

M.A. Aizerman, E.M. Braverman, and L.I. Rozoner.

Theoretical foundations of the potential function

method in pattern recognition learning.

Automation and Remote Control

, 25:821–837, 1964.

M. Anthony and N. Biggs.

Pac learning and neural networks.

The Handbook of Brain Theory and

Neural Networks

, pages 694–697, 1995.

K.P. Bennett and E. Bredensteiner.

Geometry in learning.

Geometry at Work

, page to appear,

Washington, D.C., 1998. Mathematical Association of America.

C.M. Bishop.

Neural Networks for Pattern Recognition

. Clarendon Press, Oxford, 1995.

V. Blanz, B. Sch¨

olkopf, H. B¨

ulthoﬀ, C. Burges, V. Vapnik, and T. Vetter.

Comparison of view–based

object recognition algorithms using realistic 3d models.

In C. von der Malsburg, W. von Seelen, J. C.

Vorbr¨

uggen, and B. Sendhoﬀ, editors,

Artificial Neural Networks — ICANN’96

, pages 251 – 256, Berlin,

1996. Springer Lecture Notes in Computer Science, Vol. 1112.

B. E. Boser, I. M. Guyon, and V .Vapnik.

A training algorithm for optimal margin classiﬁers.

Fifth

Annual Workshop on Computational Learning Theory

, pages 144–152, Pittsburgh, 1992. ACM.

James R. Bunch and Linda Kaufman. Some stable methods for calculating inertia and solving symmetric

linear systems.

Mathematics of computation

, 31(137):163–179, 1977.

James R. Bunch and Linda Kaufman. A computational method for the indeﬁnite quadratic programming

problem.

Linear Algebra and its Applications

, 34:341–370, 1980.

C. J. C. Burges and B. Sch¨

olkopf. Improving the accuracy and speed of support vector learning machines.

In M. Mozer, M. Jordan, and T. Petsche, editors,

Advances in Neural Information Processing Systems 9

pages 375–381, Cambridge, MA, 1997. MIT Press.

C.J.C. Burges. Simpliﬁed support vector decision rules. In Lorenza Saitta, editor,

Proceedings of the Thir-

teenth International Conference on Machine Learning

, pages 71–77, Bari, Italy, 1996. Morgan Kaufman.

C.J.C. Burges. Geometry and invariance in kernel based methods. In B. Sch¨

olkopf, C.J.C. Burges, and A.J.

Smola, editors,

Advances in Kernel Methods: Support Vector Learning

, pages 89–116. MIT Press, 1999.

C.J.C. Burges, P. Knirsch, and R. Haratsch. Support vector web page: http://svm.research.bell-labs.com.

Technical report, Lucent Technologies, 1996.

C. Cortes and V. Vapnik. Support vector networks.

Machine Learning

, 20:273–297, 1995.

R. Courant and D. Hilbert.

Methods of Mathematical Physics

. Interscience, 1953.

Luc Devroye, L´

aszl´

o Gy¨

orﬁ, and G´

abor Lugosi.

A Probabilistic Theory of Pattern Recognition

. Springer

Verlag, Applications of Mathematics Vol. 31, 1996.

H. Drucker, C.J.C. Burges, L. Kaufman, A. Smola, and V. Vapnik.

Support vector regression machines.

Advances in Neural Information Processing Systems

, 9:155–161, 1997.

R. Fletcher.

Practical Methods of Optimization

. John Wiley and Sons, Inc., 2nd edition, 1987.

S. Geman, E. Bienenstock, and R. Doursat.

Neural networks and the bias / variance dilemma.

Neural

Computation

, 4:1–58, 1992.

F. Girosi. An equivalence between sparse approximation and support vector machines.

Neural Computation

(to appear); CBCL AI Memo 1606, MIT

, 1998.

I. Guyon, V. Vapnik, B. Boser, L. Bottou, and S.A. Solla.

Structural risk minimization for character

recognition.

Advances in Neural Information Processing Systems

, 4:471–479, 1992.

P.R. Halmos.

A Hilbert Space Problem Book

. D. Van Nostrand Company, Inc., 1967.

R.A. Horn and C.R. Johnson.

Matrix Analysis

. Cambridge University Press, 1985.

T. Joachims.

Text categorization with support vector machines.

Technical report, LS VIII Number 23,

University of Dortmund, 1997. ftp://ftp-ai.informatik.uni-dortmund.de/pub/Reports/report23.ps.Z.

L. Kaufman. Solving the qp problem for support vector training. In

Proceedings of the 1997 NIPS Workshop

on Support Vector Machines (to appear)

, 1998.

A.N. Kolmogorov and S.V.Fomin.

Introductory Real Analysis

. Prentice-Hall, Inc., 1970.

O.L. Mangasarian.

Nonlinear Programming

. McGraw Hill, New York, 1969.

Garth P. McCormick.

Non Linear Programming: Theory, Algorithms and Applications

. John Wiley and

Sons, Inc., 1983.

D.C. Montgomery and E.A. Peck.

Introduction to Linear Regression Analysis

. John Wiley and Sons, Inc.,

2nd edition, 1992.

Mor´

e and Wright.

Optimization Guide

. SIAM, 1993.

Jorge J. Mor´

e and Gerardo Toraldo. On the solution of large quadratic programming problems with bound

constraints.

SIAM J. Optimization

, 1(1):93–113, 1991.

S. Mukherjee, E. Osuna, and F. Girosi. Nonlinear prediction of chaotic time series using a support vector

machine.

Proceedings of the IEEE Workshop on Neural Networks for Signal Processing 7

, pages

511–519, Amelia Island, FL, 1997.

K.-R. M¨

uller, A. Smola, G. R¨

atsch, B. Sch¨

olkopf, J. Kohlmorgen, and V. Vapnik. Predicting time series

with support vector machines. In

Proceedings, International Conference on Artificial Neural Networks

page 999. Springer Lecture Notes in Computer Science, 1997.

Edgar Osuna, Robert Freund, and Federico Girosi.

An improved training algorithm for support vector

machines. In

Proceedings of the 1997 IEEE Workshop on Neural Networks for Signal Processing, Eds.

J. Principe, L. Giles, N. Morgan, E. Wilson

, pages 276 – 285, Amelia Island, FL, 1997.

Edgar Osuna, Robert Freund, and Federico Girosi. Training support vector machines: an application to

face detection. In

IEEE Conference on Computer Vision and Pattern Recognition

, pages 130 – 136, 1997.

Edgar Osuna and Federico Girosi.

Reducing the run-time complexity of support vector machines.

International Conference on Pattern Recognition (submitted)

, 1998.

W.H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vettering.

Numerical recipes in C: the art of scientific

computing

. Cambridge University Press, 2nd edition, 1992.

M. Schmidt. Identifying speaker with support vector networks. In

Interface ’96 Proceedings

, Sydney, 1996.

B. Sch¨

olkopf.

Support Vector Learning

. R. Oldenbourg Verlag, Munich, 1997.

B. Sch¨

olkopf, C. Burges, and V. Vapnik. Extracting support data for a given task. In U. M. Fayyad and

R. Uthurusamy, editors,

Proceedings, First International Conference on Knowledge Discovery & Data

Mining

. AAAI Press, Menlo Park, CA, 1995.

B. Sch¨

olkopf, C. Burges, and V. Vapnik. Incorporating invariances in support vector learning machines. In

C. von der Malsburg, W. von Seelen, J. C. Vorbr¨

uggen, and B. Sendhoﬀ, editors,

Artificial Neural Networks

— ICANN’96

, pages 47 – 52, Berlin, 1996. Springer Lecture Notes in Computer Science, Vol. 1112.

B. Sch¨

olkopf, P. Simard, A. Smola, and V. Vapnik. Prior knowledge in support vector kernels. In M. Jordan,

M. Kearns, and S. Solla, editors,

Advances in Neural Information Processing Systems 10

, Cambridge, MA,

1998. MIT Press. In press.

B. Sch¨

olkopf, A. Smola, and K-R. Muller. Nonlinear component analysis as a kernel eigenvalue problem.

Neural Computation

, 10(5):1299–1319, 1998.

B. Sch¨

olkopf, A. Smola, K.-R. M¨

uller, C. Burges, and V. Vapnik.

Support vector methods in learning

and feature extraction.

Australian Journal of Intelligent Information Processing Systems

, 5:3 – 9, 1998.

Special issue with selected papers of ACNN’98.

B. Sch¨

olkopf, K. Sung, C. Burges, F. Girosi, P. Niyogi, T. Poggio, and V. Vapnik.

Comparing support

vector machines with gaussian kernels to radial basis function classiﬁers.

IEEE Trans. Sign. Processing

45:2758 – 2765, 1997.

John Shawe-Taylor, Peter L. Bartlett, Robert C. Williamson, and Martin Anthony.

A framework for

structural risk minimization. In

Proceedings, 9th Annual Conference on Computational Learning Theory

pages 68–76, 1996.

John Shawe-Taylor, Peter L. Bartlett, Robert C. Williamson, and Martin Anthony. Structural risk mini-

mization over data-dependent hierarchies. Technical report, NeuroCOLT Technical Report NC-TR-96-053,
1996.

A. Smola, B. Sch¨

olkopf, and K.-R. M¨

uller. General cost functions for support vector regression. In

Ninth

Australian Congress on Neural Networks (to appear)

, 1998.

A.J. Smola and B. Sch¨

olkopf. On a kernel-based method for pattern recognition, regression, approximation

and operator inversion.

Algorithmica

, 22:211 – 231, 1998.

A.J. Smola, B. Sch¨

olkopf, and K-R. M¨

uller. The connection between regularization operators and support

vector kernels.

Neural Networks

, 11:637–649, 1998.

M. O. Stitson, A. Gammerman, V. Vapnik, V.Vovk, C. Watkins, and J. Weston.

Support vector anova

decomposition. Technical report, Royal Holloway College, Report number CSD-TR-97-22, 1997.

G. Strang.

Introduction to Applied Mathematics

. Wellesley-Cambridge Press, 1986.

R. J. Vanderbei. Interior point methods : Algorithms and formulations.

ORSA J. Computing

, 6(1):32–34,

1994.

R.J. Vanderbei. LOQO: An interior point code for quadratic programming. Technical report, Program in

Statistics & Operations Research, Princeton University, 1994.

V. Vapnik.

Estimation of Dependences Based on Empirical Data [in Russian]

Nauka, Moscow, 1979.

(English translation: Springer Verlag, New York, 1982).

V. Vapnik.

The Nature of Statistical Learning Theory

. Springer-Verlag, New York, 1995.

V. Vapnik.

Statistical Learning Theory

. John Wiley and Sons, Inc., New York, 1998.

Grace Wahba. Support vector machines, reproducing kernel hilbert spaces and the gacv. In

Proceedings of

the 1997 NIPS Workshop on Support Vector Machines (to appear)

. MIT Press, 1998.

J. Weston, A. Gammerman, M. O. Stitson, V. Vapnik, V.Vovk, and C. Watkins. Density estimation using

support vector machines. Technical report, Royal Holloway College, Report number CSD-TR-97-23, 1997.