Machine Learning Notes

Optimization of hyperparameters

The problem of tuning parameters. Generally speaking, tuning is specific to a given dataset, and it is best to use a validation set or K-Fold validation. Tuning is very expensive, so it is usually best to start with the most important hyperparameters and only consider adjusting the others when necessary (good enough is good enough).

Techniques for hyperparameter tuning include grid search, random search, and Bayesian optimization.

Although some people argue that grid search handles wide parameter ranges poorly and therefore recommend Bayesian methods, individual parameters do not necessarily vary smoothly, which means that Bayesian optimization — an approach based on exploring a smooth space — is not necessarily suitable either. (It is genuinely convenient, though.)

XGBoost

The official website is always the primary source: XGBoost Hyperparameter Optimization

Core Hyperparameter List

Parameter Name (English)	Description	Effect	Recommended Value
max_depth	Maximum depth of a single tree	Greater depth increases model complexity (prone to overfitting)	3-9
min_child_weight	Minimum sum of instance weights required to split a child node	Larger values constrain tree growth (prevents overfitting)	1-7
subsample	Sampling ratio of training data used by each tree	Lower values introduce randomness (prevents overfitting)	0.6-1.0
colsample_bytree	Sampling ratio of feature columns used by each tree	Limits the feature selection space (improves generalization)	0.6-1.0
learning_rate	Shrinkage factor for the step size of weight updates	Small step sizes improve stability (requires more trees)	0.01-0.3

Ref: Suggested Ranges for Tuning XGBoost Hyperparameters

& Most Important XGBoost Hyperparameters to Tune

Extended XGBoost Hyperparameter List

Expand

Parameter Name (English)	Description	Effect	Recommended Range
gamma	Minimum loss reduction required for a node split	Larger values make the model conservative (suitable for noisy data)	0-5
reg_alpha (L1)	L1 regularization coefficient	Sparsifies feature weights (produces sparse solutions)	0-10
reg_lambda (L2)	L2 regularization coefficient	Smooths feature weights (prevents extreme values)	0-10
scale_pos_weight	Weight balancing coefficient between positive and negative samples	Handles class imbalance (boosts the positive-sample weight when the value is > 1)	1-10
n_estimators	Total number of trees in the ensemble	More trees increase model capacity (must balance computational cost)	100-2000

The parameter ranges here are basically made up (because the official website does not provide them)

---

Tuning with Optuna

Ref：Bayesian Optimization of XGBoost Hyperparameters with optuna

Example from the official website

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from xgboost import XGBClassifier
import optuna

# Generate a synthetic classification dataset
X, y = make_classification(n_samples=1000, n_classes=2, n_informative=10, random_state=42)

def objective(trial):
    # Suggest hyperparameters
    params = {
        'max_depth': trial.suggest_int('max_depth', 2, 10),
        'learning_rate': trial.suggest_float('learning_rate', 1e-3, 1.0, log=True),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
    }

    # Create an XGBoost classifier with the suggested hyperparameters
    model = XGBClassifier(**params, n_estimators=100, objective='binary:logistic', random_state=42)

    # Perform 5-fold cross-validation and return the mean accuracy
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    return scores.mean()

# Create an Optuna study with TPE sampler
study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler())

# Optimize the study for 100 trials
study.optimize(objective, n_trials=100)

# Print the best hyperparameters and best score
print(f"Best hyperparameters: {study.best_params}")
print(f"Best score: {study.best_value:.4f}")

Using manual K-Fold

from sklearn.model_selection import KFold
from xgboost import XGBRegressor, callback

def optimize_hyperparameters(target_name, X_train_, y_train_):
    """使用交叉验证进行参数优化"""
    def objective(trial):
        params = {
            'max_depth': trial.suggest_int('max_depth', 3, 9),
            'learning_rate': trial.suggest_float('learning_rate', 1e-3, 0.3, log=True),
            'subsample': trial.suggest_float('subsample', 0.6, 1.0),
            'colsample_bytree': trial.suggest_float('colsample_bytree', 0.6, 1.0),
            'gamma': trial.suggest_float('gamma', 0, 1.0),
            'reg_alpha': trial.suggest_float('reg_alpha', 0, 1.0),
            'reg_lambda': trial.suggest_float('reg_lambda', 0.5, 2.0),
            'tree_method': 'hist' #'gpu_hist'
        }
        
        # 使用分层K折交叉验证（更适合分类任务）
        kf = KFold(n_splits=5, shuffle=True, random_state=42)
        scores = []
        
        for fold_idx, (train_idx, val_idx) in enumerate(kf.split(X_train_, y_train_)):
            X_tr, y_tr = X_train_.iloc[train_idx], y_train_.iloc[train_idx]
            X_v, y_v = X_train_.iloc[val_idx], y_train_.iloc[val_idx]

            model = XGBRegressor(
                **params,
                n_estimators=2000,
                early_stopping_rounds=20,
                eval_metric='rmse',
                random_state=42 # + fold_idx,  # 为每个fold设置不同的随机种子 ?
                n_jobs=-1
            )
            
            model.fit(
                X_tr, y_tr,
                eval_set=[(X_v, y_v)],
                verbose=False
            )
            
            # 使用最佳迭代预测并计算分数
            y_pred = model.predict(X_v, iteration_range=(0, model.best_iteration + 1))
            fold_score = np.sqrt(mean_squared_error(y_v, y_pred))
            scores.append(fold_score)
            
            # 添加Optuna中途剪枝功能
            trial.report(fold_score, step=fold_idx)
            if trial.should_prune():
                raise optuna.TrialPruned()
        
        return np.mean(scores)

    # 创建带持久化存储的Optuna study
    storage_name = f"sqlite:///{target_name}_tuning_optuna.db"
    study = optuna.create_study(
        direction='minimize',
        sampler=optuna.samplers.TPESampler(seed=42),
        study_name=target_name,
        storage=storage_name,
        load_if_exists=True
    )
    
    # 优化时显示进度条并设置超时时间
    study.optimize(objective, n_trials=500, show_progress_bar=True, timeout=86400)  # 24小时超时
    
    # 保存完整的study对象
    joblib.dump(study, f"optuna_study_{target_name}.pkl")
    return study.best_params

Ps: I really want to complain about sqlite:/// — I have no idea why it is structured that way