Hard-margin vs soft-margin SVM — what's the difference?
mediumAnswer
- Hard-margin SVM finds the hyperplane that separates classes with the largest margin — only works when classes are linearly separable, otherwise there is no feasible solution.
- Soft-margin adds slack variables ≥ 0 that let some points violate the margin (xi in the margin, xi > 1 misclassified) and penalizes them via C * sum xi.
- C is a regularization knob: large C → few violations, high variance; small C → many violations, high bias.
- Almost every real SVM is soft-margin.
How to say it out loud
Hard margin assumes the classes are linearly separable and finds the widest gap between them. If they are not separable, the optimisation has no feasible solution at all — it does not degrade gracefully, it simply fails. Soft margin introduces slack variables that let points sit inside the margin or on the wrong side of it, and penalises the total slack by C. So C is the bias-variance knob: large C pushes you back towards hard margin and overfits, small C tolerates violations and underfits. In practice every SVM you will ever use is soft margin — scikit-learn does not even expose a hard-margin option, you approximate it with a very large C.
Check yourself — multiple choice
- Hard-margin is always used in practice
- Soft-margin allows margin violations penalized by C — the only realistic option for noisy / overlapping data
- Both are identical when C = 0
- Hard-margin uses kernels; soft-margin doesn't
Soft-margin = slack variables + C penalty. Hard-margin only works for perfectly separable data.
What the interviewer asks next
- How does the C parameter in an SVM affect the fit?C is where the interviewer goes next, essentially every time.
- What is a support vector, and why does the SVM only depend on them?Slack changes which points end up being support vectors, so the two ideas are linked.
- Why is feature scaling critical for SVMs?The margin is measured in feature units, so unscaled inputs quietly break everything above.
- Why do SVMs use kernels?Non-separability can be answered with slack or with a kernel, and knowing when to reach for which is the real question.
Practise Supervised Learning
215 interview questions in this topic.