Support Vector Machine: A Comprehensive Guide — Part1
Support Vector Machines (SVMs) are a type of supervised learning algorithm used for classification and regression analysis. SVMs are particularly useful when working with complex datasets with high-dimensional feature spaces, and have been successfully applied in a wide range of applications, including image classification, natural language processing, and bioinformatics.
To explain SVM I have divided this topic into 10 subtopics. I will cover only the first 5 subtopics in this article and will cover the rest in my next upcoming article.
- What is SVM?
- Equation of a Line.
- Distance of a point from a plane.
- Geometric Intuition behind SVM.
- Hard and Soft Margin.
- SVM Mathematical Intuition.
- Cost Function
- Support Vector Regression (SVR)
- SVM Kernels
- Type of SVM Kernels
What is SVM?
SVMs are based on the idea of finding the hyperplane that separates two classes in a class. In a two-dimensional feature space, the hyperplane is a line that separates the two classes. In higher dimensions, the hyperplane is a plane. The goal of SVMs is to find the hyperplane that maximizes the margin between two classes.
SVMs can solve both classification and regression problems.
- Classification → Support Vector Classifier (SVC)
- Regression → Support Vector Regressor (SVR)
Equation of Line
The equation of a line is usually written as:
y = mx + c
where m = slope of the line and c is the y-intercept.
- Slope: The slope of a line tells us with the unit movement in the independent variable(x) what is the movement in the dependent variable(y)
- Intercept: When the value of the dependent variable (x) is zero, the point where the best-fit line meets the y-axis is called intercept.
The equation of a line can also be written as follows:
Also, it can be written as:
Equation of a line passing through the origin:
Distance of a point from a plane
In the image below, we have a plane A and w is a vector perpendicular to the plane A. P1 and P2 are two points. The distance of point P1 from the plane A is d and the distance of point P2 from the plane A is d’.
As w and P1 are on the same side of the plane and are perpendicular to each other so the angle between w and P1 = 0
Therefore, the distance of point P1 from plane A is:
Here, ||w|| is the magnitude of w.
Therefore, given a plane A which has a normal passing-through origin for all points that are in half space in the direction of vector w, we get
and if the direction of points is opposite to vector w, we get
Geometric Intuition Behind SVM
In Logistic Regression we try to find the best-fit line that will separate the categories.
In Support Vector Classifier (SVC), also we try to find the best-fit line but along with that we also try to calculate two marginal planes(P1 and P2). One plane (P1) will be passing through the nearest point of one category and the other plane (P2) will be passing through the nearest point of another category such that the distance between these two marginal planes(P1 and P2) should be maximum.
The marginal planes should be equidistant from the best-fit line.
P1 is the plane parallel to Plane A and passes through the nearest positive point. P2 is the plane also parallel to plane A and passes through the nearest negative point.
As P1 is parallel to Plane A and P2 is also parallel to Plane A. Therefore, P1 and P2 are parallel to each other.
The key idea of SVM is to find a Plane A that separates positive and negative points as widely as possible. The distance between P1 and P2 is called the margin. Plane A is called Margin Maximizing Hyperplane. As the margin increases the general accuracy(accuracy on unseen data) also increases.
Points passing through P1 and P2 are called Support Vectors.
Hard Margin and Soft Margin
In Hard Margin, we will be able to separate all the points by using marginal planes and won’t get any error. On the other hand, whenever we are able to find marginal planes along with the best-fit line but with some error, we say it is a Soft Margin.
Conclusion
In this article, we discussed SVMs, the equation of a line, and the distance of a point from a plane along with the geometrical intuition of SVMs. Also, we discussed hard and soft margins. In my next article, I will try to discuss the rest of the topics mentioned above.
Thanks for reading this article! Leave a comment below if you have any questions. You can follow me on Linkedin and GitHub.
