程序代写代做代考 database jvm decision tree algorithm cache python Java In [1]:

In [1]:
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = “all”

%matplotlib inline

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style(“whitegrid”)
sns.set_context(“notebook”)
#sns.set_context(“poster”)
In [2]:
# XGBoost is not included in the Anaconda distribution (yet… )
# Therefore you need to install it first
# ! pip install xgboost
# or
# ! sudo pip install xgboost
# or
# ! pip install –upgrade xgboost
# with sudo if you don’t have admin privilegis
# in a Mac remember that you have to install Xcode and accept the license

from xgboost import XGBClassifier

from sklearn.model_selection import KFold
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score

from sklearn.metrics import accuracy_score

from sklearn import preprocessing

import xgboost as xgb

XGBoost
XGBoost stands for eXtreme Gradient Boosting.
The name XGBoost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use XGBoost. Tianqui Chen, on Quora.com (tre creator of XGBoost)
XGBoosst is an implementation of Gradient Boosting Machines created by Tianqui Chen for his PhD thesis and now expanded with contributions from many developers.
If you are interested in the story of XGBoost, Tianqui Chen explains it in the tutorial Story and Lessons Behind the Evolution of XGBoost.
XGBoost needs to be downloaded and installed in your computer (if you have a Mac you need to download XCode – the Apple development suite – and accept the license agreement first). It has interfaces to many languages besides Python, such as R, Julia, C++, Scala, Java and JVM languages, etc … XGBoost is distributed under the Apache-2 license.
XGBoost is built with a cloud platform focus in mind. In fact, AWS, Azure and Google Cloud host implementations of XGBoost tuned to their systems. Therefore XGBoost makes an extensive use of paralelization using all the cores and all the CPUs available and distributed computing for very large models. Also of techniques such out-of-core computeing for very large datasets and cache optimization to take advantage of the large cache memories in cloud servers.
Regarding the construction of the algorithm XGBoost is an implementation of gradient boosting. Gradient Boosting algorithms improve the solution building models that correct the errors of previous models. New models are created that predict the residuals or errors made by existing models. Models are added sequentially until no further improvement can be made. XGBoost supports the three main forms of gradient boosting:
• Gradient Boosting Algorithm (Gradient Boosting Machine Learning). Including the learning rate.
• Sthocastic Gradient Boosting. With sub-sampling at the row, column and column per split levels.
• Regularized Gradient Boosting. Using the L1 and L2 regularization (we’ve seen this in Ridge and Lasso regressions).
Two important additions are being sparse aware, therefore automatically supporting missing values and also supporting continuous training so you can further boost an already fitted model with new data.
One of the main reasons why XGBoost is so used is its efficiency, compared to other implementations of gradient boosting, it’s fast, memory efficient and highly accurate (check Benchmarking Random Forest Implementations by Szilard Pafka).
So far XGBoost dominates the strutured or tabular datasets on classification and regression predictive modeling problems. It’s the algorithm of choice for Kaggle competitions (XGBoost: Machine Learning Challenge Winning Solutions).
In order to be able to compare them with the previous one, we will use the same dataset, the Pima Indians.


In this exercise we will use one of the traditional Machine Learning dataset, the Pima Indians diabetes dataset.
This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset. Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage.
Content The datasets consists of several medical predictor variables and one target variable, Outcome. Predictor variables includes the number of pregnancies the patient has had, their BMI, insulin level, age, and so on.
▪ Pregnancies
▪ Glucose
▪ BloodPressure
▪ SkinThickness
▪ Insulin
▪ BMI
▪ DiabetesPedigreeFunction (scores de likelihood of diabetes based on family history)
▪ Age
▪ Outcome
In [4]:
# Load the Pima indians dataset and separate input and output components

from numpy import set_printoptions
set_printoptions(precision=3)

filename=”pima-indians-diabetes.data.csv”
names=[“pregnancies”, “glucose”, “pressure”, “skin”, “insulin”, “bmi”, “pedi”, “age”, “outcome”]
p_indians=pd.read_csv(filename, names=names)
p_indians.head()

# First we separate into input and output components
array=p_indians.values
X=array[:,0:8]
y=array[:,8]
np.set_printoptions(suppress=True)
X
pd.DataFrame(X).head()
Out[4]:

pregnancies
glucose
pressure
skin
insulin
bmi
pedi
age
outcome
0
6
148
72
35
0
33.6
0.627
50
1
1
1
85
66
29
0
26.6
0.351
31
0
2
8
183
64
0
0
23.3
0.672
32
1
3
1
89
66
23
94
28.1
0.167
21
0
4
0
137
40
35
168
43.1
2.288
33
1
Out[4]:
array([[ 6. , 148. , 72. , …, 33.6 , 0.627, 50. ],
[ 1. , 85. , 66. , …, 26.6 , 0.351, 31. ],
[ 8. , 183. , 64. , …, 23.3 , 0.672, 32. ],
…,
[ 5. , 121. , 72. , …, 26.2 , 0.245, 30. ],
[ 1. , 126. , 60. , …, 30.1 , 0.349, 47. ],
[ 1. , 93. , 70. , …, 30.4 , 0.315, 23. ]])
Out[4]:

0
1
2
3
4
5
6
7
0
6.0
148.0
72.0
35.0
0.0
33.6
0.627
50.0
1
1.0
85.0
66.0
29.0
0.0
26.6
0.351
31.0
2
8.0
183.0
64.0
0.0
0.0
23.3
0.672
32.0
3
1.0
89.0
66.0
23.0
94.0
28.1
0.167
21.0
4
0.0
137.0
40.0
35.0
168.0
43.1
2.288
33.0

XGBoost¶
XGBoost provides a wrapper to allow models to be treated like classifiers or regressors following the sckit-learn framework.
This means that you can use it in the same way that we use any other scikit-learn model.
For classification we will use the XGBClassifier class.
In [5]:
# XGBoost
# evaluated with train & test – remember we have a high variance !

seed=7
test_size=0.4

# split into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=seed)

# instantied the model
model=xgb.XGBClassifier()

# train the model on training data
model.fit(X_train, y_train)

# make predictions using tesst data
y_predict=model.predict(X_test)

# evaluate the predictions
accuracy = accuracy_score(y_test, y_predict)

print(f’XGBoost – Accuracy {accuracy*100:.3f}%’)
Out[5]:
XGBClassifier(base_score=0.5, booster=’gbtree’, colsample_bylevel=1,
colsample_bynode=1, colsample_bytree=1, gamma=0,
learning_rate=0.1, max_delta_step=0, max_depth=3,
min_child_weight=1, missing=None, n_estimators=100, n_jobs=1,
nthread=None, objective=’binary:logistic’, random_state=0,
reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
silent=None, subsample=1, verbosity=1)

XGBoost – Accuracy 78.571%
In [6]:
# XGBoost
# evaluated with KFold
# in this case we use 3 splits because the amount of data is not large

seed=7

kfold=KFold(n_splits=3, random_state=seed)

#learner=DecisionTreeClassifier(class_weight=”balanced”, random_state=seed)
learner=xgb.XGBClassifier()

results=cross_val_score(model, X, y, cv=kfold)

print(f’XGBoost with kfold – Accuracy {results.mean()*100:.3f}% std {results.std()*100:3f}’)

XGBoost with kfold – Accuracy 77.214% std 2.577993
In [7]:
# XGBoost
# evaluated with StratifiedKFold because of unbalanced classes
# in this case we use 3 splits because the amount of data is not large

seed=7

kfold=StratifiedKFold(n_splits=3, random_state=seed)

learner=xgb.XGBClassifier()

results=cross_val_score(model, X, y, cv=kfold)

print(f’XGBoost with Stratifiedkfold – Accuracy {results.mean()*100:.3f}% std {results.std()*100:3f}’)

XGBoost with Stratifiedkfold – Accuracy 77.613% std 3.074881

Plot a single XGBoost Decision Tree¶
Explainability of the algorithms is many times crucial. In this regard any tree based algorithm has a significant advantage because a tree representing the underlying structure of decisions can be plotted.
Requires the graphviz library installed.
In [8]:
# Plotting a tree

# ! pip install graphviz

from matplotlib.pylab import rcParams

##set up the parameters
rcParams[‘figure.figsize’] = 150,150

model=XGBClassifier()

model.fit(X,y)

#plotting the first tree
xgb.plot_tree(model)

#plotting the fourth tree
xgb.plot_tree(model, num_trees=4)

#plotting from left to right
xgb.plot_tree(model, num_trees=4, rankdir=”LR”)

#fig = plt.gcf()
#fig.set_size_inches(150, 150)
#fig.savefig(‘xgb_tree.png’)
Out[8]:
XGBClassifier(base_score=0.5, booster=’gbtree’, colsample_bylevel=1,
colsample_bynode=1, colsample_bytree=1, gamma=0,
learning_rate=0.1, max_delta_step=0, max_depth=3,
min_child_weight=1, missing=None, n_estimators=100, n_jobs=1,
nthread=None, objective=’binary:logistic’, random_state=0,
reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
silent=None, subsample=1, verbosity=1)
Out[8]:

Out[8]:

Out[8]:







Feature Importance¶
Similar to the Random Forest family we can obtain a vector with the relative importance of each feature and plot it.
In [9]:
# XGBoost – Feature importance

y_p=p_indians[“outcome”]
X_p=p_indians.drop([“outcome”],axis=1)

# create a train/test split
#X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(X_p, y_p, test_size=0.4, random_state=7)

plt.figure(figsize=(15,9))

seed=7

model=XGBClassifier()
model.fit(X_train,y_train)

# Feature importance is calculated as the decrease in node impurity
# weighted by the probability of reaching that node.
# The node probability can be calculated by the number of samples that reach the node,
# divided by the total number of samples. The higher the value the more important the feature.

for name, importance in zip(p_indians.columns, model.feature_importances_):
print(f'{name:15s} {importance:.4f}’)

sns.barplot(x=p_indians.columns[:-1], y=model.feature_importances_)
Out[9]: