XGBoost & SHAP Value

Background

For SHAP Value, I originally wanted to compute it with an MLP, but it turned out that tree-based models still performed best, so I went back to XGBoost. (By comparison, an MLP fits a little better. At least it's a bit more convenient.)

Calculate SHAP Value

# 创建Tree类型模型解释器
explainer = shap.TreeExplainer(
    model=model,
    data=background_data,
    feature_perturbation="interventional"
)
print(f"[info] 正在计算：{model_name} shap_values")
shap_values = explainer.shap_values(background_data)

All the code for computing and saving SHAP values

import os
import shap
import joblib
from pathlib import Path
from datetime import datetime

def save_shap_explainer(model, background_data, save_path, model_name):
    """
    生成并保存SHAP解释器
    
    参数：
        model: 训练好的模型
        background_data: 用于解释的数据
        save_path: 保存根目录
        model_name: 模型标识名称
    """
    
    save_dir = Path(save_path) / f"shap_explainers/{datetime.now().strftime('%Y%m%d')}"
    save_dir.mkdir(parents=True, exist_ok=True)
    explainer_path = save_dir / f"{model_name}_explainer.pkl"
    print(f"[info] 正在计算：{explainer_path}")
    
    # 序列化保存解释器
    if explainer_path.exists():
        print(f"[警告] 已存在文件：{explainer_path} 跳过")
        return
    else:
        # 创建Tree类型模型解释器
        explainer = shap.TreeExplainer(
            model=model,
            data=background_data,
            feature_perturbation="interventional"
        )
        print(f"[info] 正在计算：{model_name} shap_values")
        shap_values = explainer.shap_values(background_data)
        explainer_tree = shap.TreeExplainer(
            model=model,
            feature_perturbation="tree_path_dependent"
        )
        print(f"[info] 正在计算：{model_name} shap_interaction_values")
        shap_interaction_values = explainer.shap_interaction_values(background_data)
        # 创建带时间戳的保存目录
        joblib.dump((explainer,shap_values,explainer_tree,shap_interaction_values), explainer_path)
        print(f"[成功] SHAP解释器保存至：{explainer_path}")
        
# 计算主程序
background_data = shap.sample(X_train, nsamples=5000, random_state=42)
for target_name, model in loaded_models.items():
    if model is not None:
        save_shap_explainer(
            model=model,
            background_data=background_data,
            save_path="../saved_models",
            model_name=target_name
        )

Calculate SHAP interaction Value

What struck me most is the interaction values! Interaction values are quite something (they can pack a huge amount of information into them as well).

Attention!

Although it isn't stressed anywhere — perhaps because I'm not a native English speaker (so I'm not very sensitive to it) — if you want to compute interactions, you have to use this:

explainer = shap.TreeExplainer(
    model=model,
    data=None, # rather than background_data,
    feature_perturbation="tree_path_dependent" # rather than interventional,
)

Ref: Basic SHAP Interaction Value Example in XGBoost

shap issues #568

interaction_matrix = np.abs(shap_interaction_values).sum(0)
for i in range(interaction_matrix.shape[0]):
    interaction_matrix[i, i] = 0
inds = np.argsort(-interaction_matrix.sum(0))[:12]
sorted_ia_matrix = interaction_matrix[inds, :][:, inds]
pl.figure(figsize=(12, 12))
pl.imshow(sorted_ia_matrix)
pl.yticks(
    range(sorted_ia_matrix.shape[0]),
    X.columns[inds],
    rotation=50.4,
    horizontalalignment="right",
)
pl.xticks(
    range(sorted_ia_matrix.shape[0]),
    X.columns[inds],
    rotation=50.4,
    horizontalalignment="left",
)
pl.gca().xaxis.tick_top()
pl.show()

Ref: shap docs: tree_based_models (NHANES I Survival Model)