免费下载建筑图纸的网站,EDI许可证需要的网站怎么做,施坦威网站关于我们,拓之朴 做网站多少钱填充指的是利用现有知识/数据来确定缺失的数量值并填充的行为。我们有几个选择#xff0c;最常见的是用此列其余部分的均值填充缺失值#xff1b;
我们可以看到原始的数据集中有五行数据的plasma_glucose_concentration有缺失值#xff1b;
print(pima[plasma_glucose_con…填充指的是利用现有知识/数据来确定缺失的数量值并填充的行为。我们有几个选择最常见的是用此列其余部分的均值填充缺失值
我们可以看到原始的数据集中有五行数据的plasma_glucose_concentration有缺失值
print(pima[plasma_glucose_concentration].isnull().sum())
# 5我们可以先看下这5行的数据的具体情况
plasma_empty_index pima[pima[plasma_glucose_concentration].isnull()].index
print(pima.loc[plasma_empty_index])
# times_pregnant plasma_glucose_concentration diastolic_blood_pressure triceps_thickness serum_insulin bmi pedigree_function age onset_diabetes
# 75 1 None 48 20 None 24.7 0.140 22 0
# 182 1 None 74 20 23 27.7 0.299 21 0
# 342 1 None 68 35 None 32.0 0.389 22 0
# 349 5 None 80 32 None 41.0 0.346 37 1
# 502 6 None 68 41 None 39.0 0.727 41 1我们使用列的均值进行填充之后可以看到缺失值的行数为0同时可以看到5行对应字段的值都是121.686763
pima[plasma_glucose_concentration].fillna(pima[plasma_glucose_concentration].mean(), inplaceTrue)
print(pima[plasma_glucose_concentration].isnull().sum())
print(pima.loc[plasma_empty_index])# 0
# times_pregnant plasma_glucose_concentration diastolic_blood_pressure triceps_thickness serum_insulin bmi pedigree_function age onset_diabetes
# 75 1 121.686763 48 20 None 24.7 0.140 22 0
# 182 1 121.686763 74 20 23 27.7 0.299 21 0
# 342 1 121.686763 68 35 None 32.0 0.389 22 0
# 349 5 121.686763 80 32 None 41.0 0.346 37 1
# 502 6 121.686763 68 41 None 39.0 0.727 41 1我们可以直接使用sklearn的SimpleImputer来进行数据的填充
from sklearn.impute import SimpleImputerimputer SimpleImputer(strategymean)
pima_imputed imputer.fit_transform(pima)
pima_imputed pd.DataFrame(pima_imputed, columnspima_column_names)
print(pima_imputed.loc[plasma_empty_index])
# times_pregnant plasma_glucose_concentration diastolic_blood_pressure triceps_thickness serum_insulin bmi pedigree_function age onset_diabetes
# 75 1.0 121.686763 48.0 20.0 155.548223 24.7 0.140 22.0 0.0
# 182 1.0 121.686763 74.0 20.0 23.000000 27.7 0.299 21.0 0.0
# 342 1.0 121.686763 68.0 35.0 155.548223 32.0 0.389 22.0 0.0
# 349 5.0 121.686763 80.0 32.0 155.548223 41.0 0.346 37.0 1.0
# 502 6.0 121.686763 68.0 41.0 155.548223 39.0 0.727 41.0 1.0可以看到所有的列都已经填充sklearn的填充处理类确实减少了繁琐的填充工作
print(pima_imputed.isnull().sum())
# times_pregnant 0
# plasma_glucose_concentration 0
# diastolic_blood_pressure 0
# triceps_thickness 0
# serum_insulin 0
# bmi 0
# pedigree_function 0
# age 0
# onset_diabetes 0
# dtype: int64如果我们直接使用0来填充缺失值同样使用KNN模型得到的准确率是0.7357185298361768有所降低的
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCVpima_zero pima.fillna(0)
X_zero pima_zero.drop(onset_diabetes, axis1)
print(learning from {} rows.format(X_zero.shape[0]))
y_zero pima_zero[onset_diabetes]knn_params {n_neighbors:[1, 2, 3, 4, 5, 6, 7]}
knn KNeighborsClassifier()
grid GridSearchCV(knn, knn_params)
grid.fit(X_zero, y_zero)
print(grid.best_score_, grid.best_params_)
# learning from 768 rows
# 0.7357185298361768 {n_neighbors: 7}scikit-learn有一个用于构建流水线的内置模块其会对原始输入数据进行各种预处理实际的训练过程中是需要首先划分数据集如果我们在应用算法之前直接对整个数据集填充值我们就是在作弊模型其实学不到任何模式。可以看见模型的准确率是66%并不好但这不是重点。重点是训练集和测试集都是用整个X矩阵的均值填充的。这违反了机器学习流程的核心原则。当预测测试集的响应值时不能假设我们已经知道了整个数据集的均值。
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifierX pima[[serum_insulin]].copy()
y pima[onset_diabetes].copy()
entire_data_set_mean X.mean()
X X.fillna(entire_data_set_mean)
X_train, X_test, y_train, y_test train_test_split(X, y, random_state99)
knn KNeighborsClassifier()
knn.fit(X_train, y_train)
score knn.score(X_test, y_test)
print(score)
# 0.6666666666666666我们使用训练样本的均值填充训练样本和测试样本的缺失值然后进行训练
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifierX pima[[serum_insulin]].copy()
y pima[onset_diabetes].copy()
X_train, X_test, y_train, y_test train_test_split(X, y, random_state99)
X_train_mean X_train.mean()
X_train X_train.fillna(X_train_mean)
X_test X_test.fillna(X_train_mean)
knn KNeighborsClassifier()
knn.fit(X_train, y_train)
score knn.score(X_test, y_test)
print(score)
# 0.6822916666666666我们使用Pipeline结合SimpleImputer来使用均值填充数据查看训练结果
from sklearn.pipeline import Pipeline
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCVknn_para {classify__n_neighbors:[1, 2, 3, 4, 5, 6, 7]}
knn KNeighborsClassifier()
mean_impute Pipeline([(imputer, SimpleImputer(strategymean)),(classify, knn)])
X pima.drop(onset_diabetes, axis1)
y pima[onset_diabetes]
grid GridSearchCV(mean_impute, knn_para)
grid.fit(X, y)
print(grid.best_score_, grid.best_params_)
# 0.7305407011289364 {classify__n_neighbors: 7}我们使用Pipeline结合SimpleImputer来使用中位数填充数据查看训练结果
from sklearn.pipeline import Pipeline
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCVknn_para {classify__n_neighbors:[1, 2, 3, 4, 5, 6, 7]}
knn KNeighborsClassifier()
mean_impute Pipeline([(imputer, SimpleImputer(strategymedian)),(classify, knn)])
X pima.drop(onset_diabetes, axis1)
y pima[onset_diabetes]
grid GridSearchCV(mean_impute, knn_para)
grid.fit(X, y)
print(grid.best_score_, grid.best_params_)
# 0.7292589763177999 {classify__n_neighbors: 7}
