How support vector machines work
Support vector machines from the picture up: the maximum-margin line, support vectors, hinge loss and the C parameter, and how the kernel trick bends a straight boundary into a curve.
A support vector machine (SVM) is a classifier built on one geometric idea: when you draw a boundary between two classes, leave as much empty space as possible on both sides of it. The optimisation problem, the support vectors, the C parameter and the kernel trick all follow from taking that idea seriously.
This guide builds an SVM up from the picture. Every figure is drawn from a small dataset that was solved exactly, so the numbers in the text are the numbers in the diagrams.
Many lines separate the data: which is best?
Start with the simplest setting: points in two dimensions, each with one of two labels. Call the blue class +1 and the orange class −1. A linear classifier draws a straight line; points on one side are predicted blue, points on the other side orange.
Figure 1 shows eight training points and three lines. All three classify every training point correctly, so on training accuracy alone they tie. They are not equally good.
Lines A and B each pass within about 0.2 units of a training point. That is a fragile place for a boundary. Training data is a sample, and the next point from a class won’t land exactly where the previous ones did. A new orange point slightly closer to the middle than any orange point seen so far falls on the wrong side of A. A new blue point slightly below (3, 4) falls on the wrong side of B. Line C keeps 0.89 units from the nearest point on either side, so it still classifies both new points correctly.
The empty corridor between the boundary and the closest points is the margin, and the SVM’s rule is to choose the line that makes it as wide as possible. With a wide margin, small shifts in the data can’t push points across the line, and statistical learning theory backs the intuition: its error guarantees for unseen data improve as the margin grows relative to the spread of the data.
The geometry of a linear classifier
To optimise a margin we need to describe a line with numbers. A linear classifier gives every point x = (x1, x2) a score:
A positive score means blue (+1), a negative score orange (−1). The decision boundary is where the score is exactly zero, w · x + b = 0. In two dimensions that is a line, in three a plane, and in general a hyperplane. Two facts about it do most of the work:
- w is perpendicular to the boundary and points towards the positive class. Sliding along the boundary leaves the score unchanged; moving in the direction of w increases it fastest.
- The score measures distance, up to a scale. The perpendicular distance from a point x to the boundary is |w · x + b| / ‖w‖, where ‖w‖ is the length of w.
There is a catch in the second fact. Multiply w and b by 10 and the boundary doesn’t move, but every score becomes ten times bigger. To pin the scale down, SVMs adopt a convention: rescale w and b so that the points closest to the boundary score exactly +1 or −1. Those two levels define two more parallel lines, the margin lines w · x + b = +1 and w · x + b = −1, and no training point may sit between them.
Each margin line is 1/‖w‖ from the boundary, so the full width of the margin is:
Figure 2 draws all of this for the eight points from Figure 1, using line C, which in these units is w = (0.5, 1) and b = −4.5. (The next two sections show where those numbers come from.)
The last equation is the important one: a wider margin means a shorter w. Maximising the margin is the same as minimising ‖w‖, which turns a geometric goal into an optimisation problem.
Support vectors
Look again at Figure 2. Three points sit exactly on the margin lines: the orange points (1, 3) and (3, 2), and the blue point (3, 4). These are the support vectors, and they give the method its name. They hold the margin in place like tent poles; every other point is further away and plays no part in where the boundary goes.
Delete (1, 1), (4, 1), (5, 4), (4, 6) and (6, 3), retrain, and you get exactly the same w and b. Move one of them a little and nothing changes, as long as it stays out of the margin. Move a support vector and the boundary moves with it. Two consequences follow:
- The solution is sparse. A trained SVM can be described by its support vectors alone, often a small fraction of the training set. That is what keeps kernel SVMs affordable at prediction time.
- Only the boundary region matters. Points deep inside their own class have no influence at all. Logistic regression, by contrast, gives every training point some pull on the line, however far away it is.
The hard-margin optimisation problem
Put the pieces together: we want the shortest w such that every training point is on the correct side of its margin line. For a point xi with label yi, which is +1 or −1, the two cases fold into one inequality, because multiplying by the label flips the orange condition into the same shape as the blue one:
This is a quadratic programme: a quadratic objective with linear constraints. It is convex, so there is one global optimum and no local minima to get stuck in, unlike training a neural network. It is the hard-margin SVM because every constraint must hold: no point may sit inside the margin.
A worked example
For the eight points in the figures a solver returns w = (0.5, 1) and b = −4.5. You can check this by hand once you know which points are the support vectors, because each of them must score exactly ±1:
(3, 2): 3w1 + 2w2 + b = −1
(3, 4): 3w1 + 4w2 + b = +1
Subtracting the second equation from the third gives 2w2 = 2, so w2 = 1. Subtracting the second from the first gives −2w1 + w2 = 0, so w1 = 0.5. Then b = −1 − 0.5 − 3 = −4.5. The hard part, which the solver does for you, is finding out which points are the support vectors in the first place. The table checks every constraint.
| point x | label y | score f(x) = 0.5x₁ + x₂ − 4.5 | y · f(x) | status |
|---|---|---|---|---|
| (1, 1) | −1 | −3 | 3 | beyond the margin |
| (1, 3) | −1 | −1 | 1 | support vector |
| (3, 2) | −1 | −1 | 1 | support vector |
| (4, 1) | −1 | −1.5 | 1.5 | beyond the margin |
| (3, 4) | +1 | 1 | 1 | support vector |
| (5, 4) | +1 | 2 | 2 | beyond the margin |
| (4, 6) | +1 | 3.5 | 3.5 | beyond the margin |
| (6, 3) | +1 | 1.5 | 1.5 | beyond the margin |
Every y · f(x) is at least 1, so every constraint holds, and the three support vectors meet it with equality. The length of w is ‖w‖ = √(0.5² + 1²) = √1.25 ≈ 1.118, so the margin is 2 / 1.118 ≈ 1.789 wide and the nearest points are 0.894 from the boundary: the 0.89 quoted for line C in Figure 1. Because the problem is convex, no other line through this data has a wider margin.
Soft margins, slack and the C parameter
Real data is rarely this tidy. One mislabelled point can make the classes impossible to separate with a line, and then the hard-margin problem has no solution. Even when a separating line exists, a single outlier can drag it somewhere awkward.
The soft-margin SVM lets points break the rule at a price. Each point gets a slack variable ξi ≥ 0 that measures how far it sits on the wrong side of its margin line:
- ξi = 0: the point is on or beyond its margin line, as before.
- 0 < ξi ≤ 1: the point is inside the margin but not on the wrong side of the boundary.
- ξi > 1: the point is misclassified.
C is the exchange rate between a wide margin (a small ‖w‖) and a small total slack. Figure 4 shows the trade-off on 19 points, one of which is an awkward blue point at (1.4, 4.0), up among the orange ones.
With C = 100 slack is so expensive that the SVM avoids it entirely. It tilts the boundary almost flat to keep the outlier on the blue side, and the margin collapses to 0.38, held in place by just three support vectors. That is the hard-margin solution: once no point needs slack, raising C further changes nothing.
With C = 0.1 slack is cheap. The boundary keeps the orientation suggested by the bulk of the data and the margin widens to 2.86. Six points end up with non-zero slack, 2.77 in total: five sit inside the margin on their own side, and the outlier is misclassified with ξ = 1.22. In between, the margin narrows steadily as C grows; on this data it is 1.90 at C = 1 and 0.53 at C = 10. Which C generalises best is an empirical question, answered with validation data.
Small C
- Wide margin; points are allowed inside it
- More support vectors (8 in Figure 4)
- Smoother, more regularised boundary; can underfit
- Individual outliers have limited pull
Large C
- Narrow margin; few or no violations
- Fewer support vectors (3 in Figure 4)
- Fits the training set closely; can overfit
- One unusual point can swing the boundary
The same thing as a loss function: hinge loss
There is an equivalent way to write the soft-margin SVM. At the optimum each slack variable is as small as its constraint allows, so ξi = max(0, 1 − yif(xi)). Substituting that in removes the constraints:
The term in the sum is the hinge loss. The quantity y · f(x) is positive when a point is classified correctly and grows the more confidently correct it is. The hinge loss is zero once y · f(x) reaches 1 and rises linearly below that.
Read this way, an SVM is a regularised linear model: a loss on the training data plus a penalty on the size of the weights. Logistic regression has the same structure with the log loss instead of the hinge. The flat part of the hinge explains the sparsity: points with y · f(x) > 1 add nothing to the loss or its gradient, so they can’t move the boundary. It also means a linear SVM can be trained with stochastic gradient descent, which is what scikit-learn’s SGDClassifier does with its default loss="hinge".
The kernel trick
Everything so far draws straight lines, and plenty of problems need curves. Take the simplest possible case: one feature x, where the class depends on how far x is from zero. In Figure 6 the blue points lie between −1.5 and 1.4, and the orange points lie outside that range on both sides. No single threshold separates them.
The fix is to add a feature. Map each point to two numbers, φ(x) = (x, x²). The inner points have small x², the outer points large x², and in this lifted space a straight line separates them comfortably.
Training the ordinary linear SVM on the lifted points gives the line x² = 0.2x + 3.915. Translated back to the original feature, that straight line becomes two thresholds at x ≈ −1.88 and x ≈ 2.08: predict blue between them and orange outside. A linear boundary in the lifted space is a non-linear boundary in the original one.
That is the general recipe: map the data into a richer feature space where a linear separator works, then run the usual SVM there. The problem is cost. With 100 original features, the squares and pairwise products alone add 5,050 new ones, and some of the most useful feature spaces are infinite-dimensional.
Kernels: dot products without the features
This is where the structure of the SVM helps. As the next section shows, both training and prediction use the data only through dot products between pairs of points, xi · xj. After a mapping, those become φ(xi) · φ(xj). A kernel is a function that computes that dot product directly from the original inputs, without ever building φ:
For the map above, φ(x) · φ(z) = xz + x²z², so K(x, z) = xz + (xz)², computed from the raw numbers. Replacing every dot product in the SVM with a kernel is the kernel trick. The common kernels are:
- Linear: K(x, z) = x · z. No mapping; the plain SVM from the earlier sections.
- Polynomial: K(x, z) = (γ x · z + r)d. Equivalent to working with all products of features up to degree d, without listing them.
- Radial basis function (RBF, or Gaussian): K(x, z) = exp(−γ‖x − z‖²). A similarity score that is 1 when the points coincide and falls towards 0 as they move apart. Its feature space is infinite-dimensional, yet each kernel value costs one distance computation.
Not every function is a valid kernel: the matrix of kernel values on any set of points must be positive semi-definite (Mercer’s condition), which guarantees it is a dot product in some feature space.
γ: how far each point’s influence reaches
With an RBF kernel, the score at a new point is a weighted vote of the support vectors, each weighted by its similarity to the new point. γ controls how quickly that similarity falls off with distance. A small γ lets every support vector influence a wide area, so the boundary is smooth. A large γ confines each support vector to its immediate neighbourhood, so the boundary can bend around individual points.
No straight line can separate the cluster in Figure 7 from the ring around it. At γ = 0.02 the kernel is so broad that the boundary is a large, stiff oval that can’t follow the ring: it gets 84.3% of the training points and 73.1% of new points right. At γ = 0.5 the boundary is a clean loop around the cluster with 19 support vectors, and 97.2% of new points are classified correctly. At γ = 20 the boundary wraps tightly round the training points. Training accuracy is still 100%, but 66 of the 70 points have become support vectors and new-data accuracy drops to 89.4%. That is overfitting, the RBF version of setting C too high.
γ and C both control flexibility, so they are tuned together. And because γ multiplies a squared distance, its sensible range depends on the scale of the features.
The dual form, briefly
The kernel trick relies on the claim that an SVM only touches the data through dot products. That comes from rewriting the optimisation problem in its dual form with Lagrange multipliers. The derivation is optional; the result is worth knowing.
Each training point gets a multiplier αi ≥ 0. At the optimum, the weight vector is a weighted sum of the training points:
and the multipliers solve a problem in which the points appear only in pairwise dot products:
Substituting w into f(x), and replacing each dot product with a kernel, gives the general prediction rule:
Three things follow:
- Only dot products appear, xi · xj in training and xi · x in prediction, so swapping in K is all the kernel trick needs.
- Only support vectors have αi > 0. The optimality conditions force αi = 0 for every point strictly beyond its margin line. Points with slack sit at the upper limit, αi = C.
- Prediction cost grows with the number of support vectors, not with the size of the training set.
For the hard-margin example, the solver gives α = 0.25 for (1, 3), α = 0.375 for (3, 2), α = 0.625 for (3, 4), and α = 0 for the other five points. Check: 0.625 · (3, 4) − 0.25 · (1, 3) − 0.375 · (3, 2) = (0.5, 1), which is w, and 0.625 − 0.25 − 0.375 = 0, as the constraint Σ αiyi = 0 requires.
With an RBF kernel the formula has a direct picture. Each support vector contributes a bump centred on itself, scaled by αi and pointing up or down according to its class. The prediction is the sum of the bumps plus b. Figure 8 shows this for the 1D data from Figure 6, now with an RBF kernel and no explicit feature map at all.
Seven of the eleven points are support vectors. The sum crosses zero at x ≈ −1.85 and x ≈ 1.91, close to the thresholds the explicit (x, x²) map produced, even though the RBF kernel was never told anything about squares.
Using SVMs in practice
Scale the features
SVMs work with dot products and distances, so a feature measured in thousands swamps one measured in fractions. Standardise every feature to mean 0 and standard deviation 1 (scikit-learn’s StandardScaler), and fit the scaler inside a pipeline so that during cross-validation it only ever sees the training folds.
Tune C and γ together
The RBF kernel is a sensible default. Search C and γ on a logarithmic grid (for example 0.1, 1, 10, 100) with cross-validation, then confirm the winner once on a held-out test set. scikit-learn’s default, gamma="scale", sets γ to 1 / (number of features × variance of the data), which is a reasonable starting point after standardising. For very high-dimensional sparse data such as text, try a linear kernel first: the data is often close to linearly separable already.
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=0
)
pipe = Pipeline([
("scale", StandardScaler()), # fitted on each training fold only
("svm", SVC(kernel="rbf")),
])
param_grid = {
"svm__C": [0.1, 1, 10, 100],
"svm__gamma": ["scale", 0.001, 0.01, 0.1, 1],
}
search = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
search.fit(X_train, y_train)
print(search.best_params_) # e.g. {'svm__C': 10, 'svm__gamma': 0.01}
print(search.best_score_) # mean cross-validated accuracy
print(search.score(X_test, y_test)) # one honest check on held-out dataMore than two classes
An SVM is a binary classifier. For k classes there are two standard reductions. One-vs-rest trains k classifiers, each separating one class from all the others, and picks the class with the highest score. One-vs-one trains k(k − 1)/2 classifiers, one per pair of classes, and takes a vote. scikit-learn’s SVC uses one-vs-one internally, while LinearSVC uses one-vs-rest.
Scores are not probabilities
f(x) is a score, not a probability. Setting probability=True on SVC fits a logistic curve to the scores (Platt scaling) with an internal cross-validation, which slows training and can disagree with predict on borderline points. For well-calibrated probabilities, use CalibratedClassifierCV and check the result on held-out data.
Watch the dataset size
Kernel SVM training works with kernel values between pairs of points, and its run time grows at least quadratically with the number of samples, so it gets slow beyond tens of thousands of rows. Use a linear SVM (LinearSVC or SGDClassifier), or approximate the kernel with explicit random features (Nystroem, RBFSampler) and train a linear model on top.
Summary
- An SVM picks the separating boundary with the widest margin to the nearest training points.
- A linear classifier scores points with f(x) = w · x + b. Fixing the closest points at scores of ±1 makes the margin 2/‖w‖, so maximising the margin means minimising ½‖w‖².
- Only the support vectors, the points on or inside the margin, determine the solution.
- The hard-margin problem is a convex quadratic programme with one global optimum.
- The soft margin adds slack ξi priced by C. Small C means a wide, tolerant margin; large C means a narrow one that chases every point. Equivalently, an SVM minimises hinge loss plus an L2 penalty.
- The kernel trick replaces dot products with K(x, z) = φ(x) · φ(z), giving curved boundaries without building the feature map. With the RBF kernel, too small a γ underfits and too large overfits.
- In the dual form, f(x) = Σ αiyiK(xi, x) + b, and only support vectors have αi > 0.
- Standardise features, tune C and γ together with cross-validation, and switch to linear or approximate methods for large datasets.
Further reading on this site
- How backpropagation works, step by step: how neural networks are trained by gradient descent on a non-convex loss, the main alternative to the convex optimisation behind SVMs.
- How convolution works in a CNN: before CNNs, image classifiers often fed hand-designed features into an SVM; convolutional layers learn the features instead.
- How retrieval-augmented generation works: covers embeddings, the fixed-length vectors that a linear SVM can classify quickly.