Problem 1¶
Load Iris data. Estimate the SVM classifier of three types of Iris using hard margin SVM (C = Inf) cross-validation = 3. Use the code below.For fit measure use default cross-validation score — share correct predictions. Can you estimate the model? If you cannot why?
In [1]:
from __future__ import division, print_function, unicode_literals
# Common imports
import numpy as np
import os
# to make this notebook’s output stable across runs
np.random.seed(42)
# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams[‘axes.labelsize’] = 14
plt.rcParams[‘xtick.labelsize’] = 12
plt.rcParams[‘ytick.labelsize’] = 12
In [2]:
from sklearn.svm import SVC
# import datasets
from sklearn import datasets
iris = datasets.load_iris()
X = iris[“data”][:,(1,2)] # Use all X
y = iris[“target”]
virginica_or_versicolor = (y == 2) | (y == 1)
X = X[virginica_or_versicolor]
y = y[virginica_or_versicolor]
In [ ]:
from sklearn.model_selection import cross_val_score
svm_clf = SVC(kernel=”linear”, C=float(“inf”))
scores = cross_val_score(svm_clf, X, y, cv=5)
np.mean(scores)
Problem 2¶
Run the same model with a soft margin: C = 1 and plot decision boundary. Don’t forget to use plt.plot(X[:, 0][y==2], X[:, 1][y==2], “ms”) when plotting instead of plt.plot(X[:, 0][y==0], X[:, 1][y==0], “ms”). Describe the results in Problem 2 and 1. Why do we get these results?
In [ ]:
In [ ]:
def plot_svc_decision_boundary(svm_clf, xmin, xmax):
# Get plot and intercept
w = svm_clf.coef_[0]
b = svm_clf.intercept_[0]
# At the decision boundary, w0*x0 + w1*x1 + b = 0
# we predict positive or negative. The formula for the line is:
# => x1 = -w0/w1 * x0 – b/w1
x0 = np.linspace(xmin, xmax, 200)
# line as a function of x0
decision_boundary = -w[0]/w[1] * x0 – b/w[1]
margin = 1/w[1]# add margin
gutter_up = decision_boundary + margin
gutter_down = decision_boundary – margin
svs = svm_clf.support_vectors_
plt.scatter(svs[:, 0], svs[:, 1], s=180, facecolors=’#FFAAAA’)
plt.plot(x0, decision_boundary, “k-“, linewidth=2)
plt.plot(x0, gutter_up, “k–“, linewidth=2)
plt.plot(x0, gutter_down, “k–“, linewidth=2)
plt.figure(figsize=(12,3.2))
plt.subplot(121)
plt.plot(X[:, 0][y==1], X[:, 1][y==1], “bo”)
plt.plot(X[:, 0][y==2], X[:, 1][y==2], “ms”)
plot_svc_decision_boundary(svm_clf, 0, 6)
plt.xlabel(“$x_0$”, fontsize=20)
plt.ylabel(“$x_1$ “, fontsize=20, rotation=0)
plt.title(“Unscaled”, fontsize=16)
Problem 3:¶
Reestimate model in problem 2 with a) optimal C parameter. Loop through a vector 1000 random numbers from .01 to 10. b) scaled data and C = 1. Which produces better cross-validation score?
In [ ]:
np.random.seed(42)
cvec = np.random.uniform(0.01,100,1000)
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
Problem 4¶
Use full data for Irises. Use scaled data created below. Estimate baseline SVM model with C = 1, calculate cross-validation score with cv =5. The estimate the same model with 5th degree polynomial of X. Report averages for both models.
In [ ]:
from sklearn.svm import SVC
# import datasets
from sklearn.preprocessing import PolynomialFeatures
from sklearn import datasets
iris = datasets.load_iris()
X = iris[“data”]
y = iris[“target”]
X_sc = scaler.fit_transform(X)
poly = PolynomialFeatures(5)
X2 = poly.fit_transform(X)
X_poly = scaler.fit_transform(X2)
In [ ]:
In [ ]:
Problem 5¶
Create a datasets of moons below. Estimate the SVM model using 10 degress polynomial and C =1. Create polynomials in two ways: a) use PolynomialFeatures(degree=10), b) use kernel SVC(kernel=”poly”, degree=10, coef0=1, C=1). Use cross-validation cv =5 and report average score. Measure the time of using a kernel relatively to the polynomial features.
In [ ]:
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=10000, noise=0.25, random_state=42)
In [ ]:
def plot_dataset(X, y, axes):
plt.plot(X[:, 0][y==0], X[:, 1][y==0], “bs”)
plt.plot(X[:, 0][y==1], X[:, 1][y==1], “g^”)
plt.axis(axes)
plt.grid(True, which=’both’)
plt.xlabel(r”$x_1$”, fontsize=20)
plt.ylabel(r”$x_2$”, fontsize=20, rotation=0)
plot_dataset(X, y, [-2, 3, -2, 2])
plt.show()
In [ ]:
In [ ]:
In [ ]:
In [ ]:
Problem 6¶
Run gaussian Kernel RBF with 10 degrees. Predict Y using cross-validation cv=5 and C =1. Report average score. Is it higher than the 10-degree polinomial?
In [ ]:
np.random.seed(42)
svm_rbf = SVC(kernel=”rbf”, degree = 10, gamma=1, C=1)
In [ ]:
In [ ]: