本文是中國大學(xué)慕課《機器學(xué)習(xí)》的“集成學(xué)習(xí)”章節(jié)的課后代碼。

課程地址：

(相關(guān)資料圖)

https://www.icourse163.org/course/WZU-1464096179

課程完整代碼：

https://github.com/fengdu78/WZU-machine-learning-course

代碼修改并注釋：黃海廣，haiguang2000@wzu.edu.cn

importwarningswarnings.filterwarnings("ignore")importpandasaspdfromsklearn.model_selectionimporttrain_test_split

生成12000行的數(shù)據(jù)，訓(xùn)練集和測試集按照3:1劃分

fromsklearn.datasetsimportmake_hastie_10_2data,target=make_hastie_10_2()

X_train,X_test,y_train,y_test=train_test_split(data,target,random_state=123)X_train.shape,X_test.shape

((9000, 10), (3000, 10))

對比六大模型，都使用默認(rèn)參數(shù)

fromsklearn.linear_modelimportLogisticRegressionfromsklearn.ensembleimportRandomForestClassifierfromsklearn.ensembleimportAdaBoostClassifierfromsklearn.ensembleimportGradientBoostingClassifierfromxgboostimportXGBClassifierfromlightgbmimportLGBMClassifierfromsklearn.model_selectionimportcross_val_scoreimporttimeclf1=LogisticRegression()clf2=RandomForestClassifier()clf3=AdaBoostClassifier()clf4=GradientBoostingClassifier()clf5=XGBClassifier()clf6=LGBMClassifier()forclf,labelinzip([clf1,clf2,clf3,clf4,clf5,clf6],["LogisticRegression","RandomForest","AdaBoost","GBDT","XGBoost","LightGBM"]):start=time.time()scores=cross_val_score(clf,X_train,y_train,scoring="accuracy",cv=5)end=time.time()running_time=end-startprint("Accuracy:%0.8f (+/-%0.2f),耗時%0.2f秒。模型名稱[%s]"%(scores.mean(),scores.std(),running_time,label))

Accuracy: 0.47488889 (+/- 0.00),耗時0.04秒。模型名稱[Logistic Regression]Accuracy: 0.88966667 (+/- 0.01),耗時16.34秒。模型名稱[Random Forest]Accuracy: 0.88311111 (+/- 0.00),耗時3.39秒。模型名稱[AdaBoost]Accuracy: 0.91388889 (+/- 0.01),耗時13.14秒。模型名稱[GBDT]Accuracy: 0.92977778 (+/- 0.00),耗時3.60秒。模型名稱[XGBoost]Accuracy: 0.93188889 (+/- 0.01),耗時0.58秒。模型名稱[LightGBM]

對比了六大模型，可以看出，邏輯回歸速度最快，但準(zhǔn)確率最低。而LightGBM，速度快，而且準(zhǔn)確率最高，所以，現(xiàn)在處理結(jié)構(gòu)化數(shù)據(jù)的時候，大部分都是用LightGBM算法。

importxgboostasxgb#記錄程序運行時間importtimestart_time=time.time()#xgb矩陣賦值xgb_train=xgb.DMatrix(X_train,y_train)xgb_test=xgb.DMatrix(X_test,label=y_test)##參數(shù)params={"booster":"gbtree",#"silent":1,#設(shè)置成1則沒有運行信息輸出，最好是設(shè)置為0.#"nthread":7,#cpu線程數(shù)默認(rèn)最大"eta":0.007,#如同學(xué)習(xí)率"min_child_weight":3,#這個參數(shù)默認(rèn)是1，是每個葉子里面h的和至少是多少，對正負(fù)樣本不均衡時的0-1分類而言#，假設(shè) h 在0.01 附近，min_child_weight 為 1 意味著葉子節(jié)點中最少需要包含 100個樣本。#這個參數(shù)非常影響結(jié)果，控制葉子節(jié)點中二階導(dǎo)的和的最小值，該參數(shù)值越小，越容易 overfitting。"max_depth":6,#構(gòu)建樹的深度，越大越容易過擬合"gamma":0.1,#樹的葉子節(jié)點上作進(jìn)一步分區(qū)所需的最小損失減少,越大越保守，一般0.1、0.2這樣子。"subsample":0.7,#隨機采樣訓(xùn)練樣本"colsample_bytree":0.7,#生成樹時進(jìn)行的列采樣"lambda":2,#控制模型復(fù)雜度的權(quán)重值的L2正則化項參數(shù)，參數(shù)越大，模型越不容易過擬合。#"alpha":0,#L1正則項參數(shù)#"scale_pos_weight":1, #如果取值大于0的話，在類別樣本不平衡的情況下有助于快速收斂。#"objective":"multi:softmax",#多分類的問題#"num_class":10,#類別數(shù)，多分類與multisoftmax并用"seed":1000,#隨機種子#"eval_metric":"auc"}plst=list(params.items())num_rounds=500#迭代次數(shù)watchlist=[(xgb_train,"train"),(xgb_test,"val")]

#訓(xùn)練模型并保存#early_stopping_rounds當(dāng)設(shè)置的迭代次數(shù)較大時，early_stopping_rounds可在一定的迭代次數(shù)內(nèi)準(zhǔn)確率沒有提升就停止訓(xùn)練model=xgb.train(plst,xgb_train,num_rounds,watchlist,early_stopping_rounds=100,)#model.save_model("./model/xgb.model")#用于存儲訓(xùn)練出的模型print("bestbest_ntree_limit",model.best_ntree_limit)y_pred=model.predict(xgb_test,ntree_limit=model.best_ntree_limit)print("error=%f"%(sum(1foriinrange(len(y_pred))ifint(y_pred[i]>0.5)!=y_test[i])/float(len(y_pred))))#輸出運行時長cost_time=time.time()-start_timeprint("xgboostsuccess!","\n","costtime:",cost_time,"(s)......")

[0]train-rmse:1.11000val-rmse:1.10422[1]train-rmse:1.10734val-rmse:1.10182[2]train-rmse:1.10465val-rmse:1.09932[3]train-rmse:1.10207val-rmse:1.09694

……

[497]train-rmse:0.62135val-rmse:0.68680[498]train-rmse:0.62096val-rmse:0.68650[499]train-rmse:0.62056val-rmse:0.68624best best_ntree_limit 500error=0.826667xgboost success!  cost time: 3.5742645263671875 (s)......

會改變的函數(shù)名是：

eta -> learning_rate

lambda -> reg_lambda

alpha -> reg_alpha

fromsklearn.model_selectionimporttrain_test_splitfromsklearnimportmetricsfromxgboostimportXGBClassifierclf=XGBClassifier(# silent=0, #設(shè)置成1則沒有運行信息輸出，最好是設(shè)置為0.是否在運行升級時打印消息。#nthread=4,#cpu線程數(shù)默認(rèn)最大learning_rate=0.3,#如同學(xué)習(xí)率min_child_weight=1,#這個參數(shù)默認(rèn)是1，是每個葉子里面h的和至少是多少，對正負(fù)樣本不均衡時的0-1分類而言#，假設(shè) h 在0.01 附近，min_child_weight 為 1 意味著葉子節(jié)點中最少需要包含 100個樣本。#這個參數(shù)非常影響結(jié)果，控制葉子節(jié)點中二階導(dǎo)的和的最小值，該參數(shù)值越小，越容易 overfitting。max_depth=6,#構(gòu)建樹的深度，越大越容易過擬合gamma=0,#樹的葉子節(jié)點上作進(jìn)一步分區(qū)所需的最小損失減少,越大越保守，一般0.1、0.2這樣子。subsample=1,#隨機采樣訓(xùn)練樣本訓(xùn)練實例的子采樣比max_delta_step=0,#最大增量步長，我們允許每個樹的權(quán)重估計。colsample_bytree=1,#生成樹時進(jìn)行的列采樣reg_lambda=1,#控制模型復(fù)雜度的權(quán)重值的L2正則化項參數(shù)，參數(shù)越大，模型越不容易過擬合。#reg_alpha=0,#L1正則項參數(shù)#scale_pos_weight=1, #如果取值大于0的話，在類別樣本不平衡的情況下有助于快速收斂。平衡正負(fù)權(quán)重#objective="multi:softmax",#多分類的問題指定學(xué)習(xí)任務(wù)和相應(yīng)的學(xué)習(xí)目標(biāo)#num_class=10,#類別數(shù)，多分類與multisoftmax并用n_estimators=100,#樹的個數(shù)seed=1000#隨機種子#eval_metric="auc")clf.fit(X_train,y_train)y_true,y_pred=y_test,clf.predict(X_test)print("Accuracy:%.4g"%metrics.accuracy_score(y_true,y_pred))

Accuracy : 0.936

importlightgbmaslgbfromsklearn.metricsimportmean_squared_error#加載你的數(shù)據(jù)#print("Loaddata...")#df_train=pd.read_csv("../regression/regression.train",header=None,sep="\t")#df_test=pd.read_csv("../regression/regression.test",header=None,sep="\t")##y_train=df_train[0].values#y_test=df_test[0].values#X_train=df_train.drop(0,axis=1).values#X_test=df_test.drop(0,axis=1).values#創(chuàng)建成lgb特征的數(shù)據(jù)集格式lgb_train=lgb.Dataset(X_train,y_train)#將數(shù)據(jù)保存到LightGBM二進(jìn)制文件將使加載更快lgb_eval=lgb.Dataset(X_test,y_test,reference=lgb_train)#創(chuàng)建驗證數(shù)據(jù)#將參數(shù)寫成字典下形式params={"task":"train","boosting_type":"gbdt",#設(shè)置提升類型"objective":"regression",#目標(biāo)函數(shù)"metric":{"l2","auc"},#評估函數(shù)"num_leaves":31,#葉子節(jié)點數(shù)"learning_rate":0.05,#學(xué)習(xí)速率"feature_fraction":0.9,#建樹的特征選擇比例"bagging_fraction":0.8,#建樹的樣本采樣比例"bagging_freq":5,#k意味著每k次迭代執(zhí)行bagging"verbose":1#<0顯示致命的,=0顯示錯誤(警告),>0顯示信息}print("Starttraining...")#訓(xùn)練cvandtraingbm=lgb.train(params,lgb_train,num_boost_round=500,valid_sets=lgb_eval,early_stopping_rounds=5)#訓(xùn)練數(shù)據(jù)需要參數(shù)列表和數(shù)據(jù)集print("Savemodel...")gbm.save_model("model.txt")#訓(xùn)練后保存模型到文件print("Startpredicting...")#預(yù)測數(shù)據(jù)集y_pred=gbm.predict(X_test,num_iteration=gbm.best_iteration)#如果在訓(xùn)練期間啟用了早期停止，可以通過best_iteration方式從最佳迭代中獲得預(yù)測#評估模型print("error=%f"%(sum(1foriinrange(len(y_pred))ifint(y_pred[i]>0.5)!=y_test[i])/float(len(y_pred))))

Start training...[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000448 seconds.You can set `force_col_wise=true` to remove the overhead.[LightGBM] [Info] Total Bins 2550[LightGBM] [Info] Number of data points in the train set: 9000, number of used features: 10[LightGBM] [Info] Start training from score 0.012000[1]valid_0"s auc: 0.814399valid_0"s l2: 0.965563Training until validation scores don"t improve for 5 rounds[2]valid_0"s auc: 0.84729valid_0"s l2: 0.934647[3]valid_0"s auc: 0.872805valid_0"s l2: 0.905265[4]valid_0"s auc: 0.884117valid_0"s l2: 0.877875[5]valid_0"s auc: 0.895115valid_0"s l2: 0.852189

……

[191]valid_0"s auc: 0.982783valid_0"s l2: 0.319851[192]valid_0"s auc: 0.982751valid_0"s l2: 0.319971[193]valid_0"s auc: 0.982685valid_0"s l2: 0.320043Early stopping, best iteration is:[188]valid_0"s auc: 0.982794valid_0"s l2: 0.319746Save model...Start predicting...error=0.664000

fromsklearnimportmetricsfromlightgbmimportLGBMClassifierclf=LGBMClassifier(boosting_type="gbdt",#提升樹的類型gbdt,dart,goss,rfnum_leaves=31,#樹的最大葉子數(shù)，對比xgboost一般為2^(max_depth)max_depth=-1,#最大樹的深度learning_rate=0.1,#學(xué)習(xí)率n_estimators=100,#擬合的樹的棵樹，相當(dāng)于訓(xùn)練輪數(shù)subsample_for_bin=200000,objective=None,class_weight=None,min_split_gain=0.0,#最小分割增益min_child_weight=0.001,#分支結(jié)點的最小權(quán)重min_child_samples=20,subsample=1.0,#訓(xùn)練樣本采樣率行subsample_freq=0,#子樣本頻率colsample_bytree=1.0,#訓(xùn)練特征采樣率列reg_alpha=0.0,#L1正則化系數(shù)reg_lambda=0.0,#L2正則化系數(shù)random_state=None,n_jobs=-1,silent=True,)clf.fit(X_train,y_train,eval_metric="auc")#設(shè)置驗證集合verbose=False不打印過程clf.fit(X_train,y_train)y_true,y_pred=y_test,clf.predict(X_test)print("Accuracy:%.4g"%metrics.accuracy_score(y_true,y_pred))

Accuracy : 0.927

1.https://xgboost.readthedocs.io/

2.https://lightgbm.readthedocs.io/

3.https://blog.csdn.net/q383700092/article/details/53763328?locationNum=9&fps=1

往期精彩回顧適合初學(xué)者入門人工智能的路線及資料下載(圖文+視頻)機器學(xué)習(xí)入門系列下載機器學(xué)習(xí)及深度學(xué)習(xí)筆記等資料打印《統(tǒng)計學(xué)習(xí)方法》的代碼復(fù)現(xiàn)專輯機器學(xué)習(xí)交流qq群955171419，加入微信群請掃碼

關(guān)鍵詞：