虽然一些基于概率的机器学习模型（如朴素贝叶斯）对特征独立性做出了大胆的假设，但逻辑回归采取了更为慎重的方法。可以将其视为绘制一条分隔两个结果的线（或平面），这使我们能够更灵活地预测概率。

定义

逻辑回归是一种用于预测二元结果的统计方法。尽管它的名字如此，但它用于分类而不是回归。它估计一个实例属于某个特定类别的概率。如果估计的概率大于 50%，则模型预测该实例属于该类别（反之亦然）。

使用的数据集

在本文中，我们将使用这个人工高尔夫数据集作为示例。该数据集根据天气状况预测一个人是否会打高尔夫球。

就像在 KNN 中一样，逻辑回归需要先缩放数据。将分类列转换为 0 和 1，并缩放数值特征，以便没有任何单个特征主导距离度量。

列：'Outlook'、'Temperature'、'Humidity'、'Wind' 和 'Play'（目标特征）。分类列（Outlook 和 Windy）使用独热编码进行编码，而数值列使用标准缩放（z 规范化）进行缩放。

# Import required libraries
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np

# Create dataset from dictionary
dataset_dict = {
    'Outlook': ['sunny', 'sunny', 'overcast', 'rainy', 'rainy', 'rainy', 'overcast', 'sunny', 'sunny', 'rainy', 'sunny', 'overcast', 'overcast', 'rainy', 'sunny', 'overcast', 'rainy', 'sunny', 'sunny', 'rainy', 'overcast', 'rainy', 'sunny', 'overcast', 'sunny', 'overcast', 'rainy', 'overcast'],
    'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0, 72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0, 88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
    'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0, 90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0, 65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
    'Wind': [False, True, False, False, False, True, True, False, False, False, True, True, False, True, True, False, False, True, False, True, True, False, True, False, False, True, False, False],
    'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
}
df = pd.DataFrame(dataset_dict)

# Prepare data: encode categorical variables
df = pd.get_dummies(df, columns=['Outlook'], prefix='', prefix_sep='', dtype=int)
df['Wind'] = df['Wind'].astype(int)
df['Play'] = (df['Play'] == 'Yes').astype(int)

# Rearrange columns
column_order = ['sunny', 'overcast', 'rainy', 'Temperature', 'Humidity', 'Wind', 'Play']
df = df[column_order]

# Split data into features and target
X, y = df.drop(columns='Play'), df['Play']

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)

# Scale numerical features
scaler = StandardScaler()
X_train[['Temperature', 'Humidity']] = scaler.fit_transform(X_train[['Temperature', 'Humidity']])
X_test[['Temperature', 'Humidity']] = scaler.transform(X_test[['Temperature', 'Humidity']])

# Print results
print("Training set:")
print(pd.concat([X_train, y_train], axis=1), '\n')
print("Test set:")
print(pd.concat([X_test, y_test], axis=1))

主要机制

逻辑回归的工作原理是将逻辑函数应用于输入特征的线性组合。它的运作方式如下：

1. 计算输入特征的加权和（类似于线性回归）。
2. 将逻辑函数（也称为 S 型函数）应用于该总和，将任何实数映射到 0 到 1 之间的值。
3. 将该值解释为属于正类的概率。
4. 使用阈值（通常为 0.5）做出最终的分类决定。

对于我们的高尔夫数据集，逻辑回归可能会将天气因素组合成一个分数，然后将该分数转换为打高尔夫的概率。

训练步骤

逻辑回归的训练过程涉及找到输入特征的最佳权重。以下是总体概述：

1. 初始化权重（通常为较小的随机值）。

# Initialize weights (including bias) to 0.1
initial_weights = np.full(X_train_np.shape[1], 0.1)

# Create and display DataFrame for initial weights
print(f"Initial Weights: {initial_weights}")

2. 对于每个训练示例：
a. 使用当前权重计算预测概率。

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def calculate_probabilities(X, weights):
    z = np.dot(X, weights)
    return sigmoid(z)

def calculate_log_loss(probabilities, y):
    return -y * np.log(probabilities) - (1 - y) * np.log(1 - probabilities)

def create_output_dataframe(X, y, weights):
    probabilities = calculate_probabilities(X, weights)
    log_losses = calculate_log_loss(probabilities, y)
    
    df = pd.DataFrame({
        'Probability': probabilities,
        'Label': y,
        'Log Loss': log_losses
    })
    
    return df

def calculate_average_log_loss(X, y, weights):
    probabilities = calculate_probabilities(X, weights)
    log_losses = calculate_log_loss(probabilities, y)
    return np.mean(log_losses)

# Convert X_train and y_train to numpy arrays for easier computation
X_train_np = X_train.to_numpy()
y_train_np = y_train.to_numpy()

# Add a column of 1s to X_train_np for the bias term
X_train_np = np.column_stack((np.ones(X_train_np.shape[0]), X_train_np))

# Create and display DataFrame for initial weights
initial_df = create_output_dataframe(X_train_np, y_train_np, initial_weights)
print(initial_df.to_string(index=False, float_format=lambda x: f"{x:.6f}"))
print(f"\nAverage Log Loss: {calculate_average_log_loss(X_train_np, y_train_np, initial_weights):.6f}")

b. 通过计算对数损失将该概率与实际类标签进行比较。

3. 更新权重以最小化损失（通常使用一些优化算法，如梯度下降。这包括重复执行步骤 2，直到对数损失无法变小）。

def gradient_descent_step(X, y, weights, learning_rate):
    m = len(y)
    probabilities = calculate_probabilities(X, weights)
    gradient = np.dot(X.T, (probabilities - y)) / m
    new_weights = weights - learning_rate * gradient  # Create new array for updated weights
    return new_weights

# Perform one step of gradient descent (one of the simplest optimization algorithm)
learning_rate = 0.1
updated_weights = gradient_descent_step(X_train_np, y_train_np, initial_weights, learning_rate)

# Print initial and updated weights
print("\nInitial weights:")
for feature, weight in zip(['Bias'] + list(X_train.columns), initial_weights):
    print(f"{feature:11}: {weight:.2f}")

print("\nUpdated weights after one iteration:")
for feature, weight in zip(['Bias'] + list(X_train.columns), updated_weights):
    print(f"{feature:11}: {weight:.2f}")    

# With sklearn, you can get the final weights (coefficients)
# and final bias (intercepts) easily.
# The result is almost the same as doing it manually above.

from sklearn.linear_model import LogisticRegression

lr_clf = LogisticRegression(penalty=None, solver='saga')
lr_clf.fit(X_train, y_train)

coefficients = lr_clf.coef_
intercept = lr_clf.intercept_

y_train_prob = lr_clf.predict_proba(X_train)[:, 1]
loss = -np.mean(y_train * np.log(y_train_prob) + (1 - y_train) * np.log(1 - y_train_prob))

print(f"Weights & Bias Final: {coefficients[0].round(2)}, {round(intercept[0],2)}")
print("Loss Final:", loss.round(3))

分类步骤

模型训练完成后：
1. 对于新实例，使用最终权重（也称为系数）计算概率，就像在训练步骤中一样。

2. 通过查看概率来解释输出：如果p ≥ 0.5，则预测为 1 类；否则，预测为 0 类

# Calculate prediction probability
predicted_probs = lr_clf.predict_proba(X_test)[:, 1]

z_values = np.log(predicted_probs / (1 - predicted_probs))

result_df = pd.DataFrame({
    'ID': X_test.index,
    'Z-Values': z_values.round(3),
    'Probabilities': predicted_probs.round(3)
}).set_index('ID')

print(result_df)

# Make predictions
y_pred = lr_clf.predict(X_test)
print(y_pred)

评估步骤

result_df = pd.DataFrame({
    'ID': X_test.index,
    'Label': y_test,
    'Probabilities': predicted_probs.round(2),
    'Prediction': y_pred,
}).set_index('ID')

print(result_df)

关键参数

逻辑回归有几个控制其行为的重要参数：

1.惩罚：要使用的正则化类型（“l1”、“l2”、“elasticnet”或“none”）。逻辑回归中的正则化通过向模型的损失函数添加惩罚项来防止过度拟合，从而鼓励更简单的模型。

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

regs = [None, 'l1', 'l2']
coeff_dict = {}

for reg in regs:
    lr_clf = LogisticRegression(penalty=reg, solver='saga')
    lr_clf.fit(X_train, y_train)
    coefficients = lr_clf.coef_
    intercept = lr_clf.intercept_
    predicted_probs = lr_clf.predict_proba(X_train)[:, 1]
    loss = -np.mean(y_train * np.log(predicted_probs) + (1 - y_train) * np.log(1 - predicted_probs))
    predictions = lr_clf.predict(X_test)
    accuracy = accuracy_score(y_test, predictions)

    coeff_dict[reg] = {
        'Coefficients': coefficients,
        'Intercept': intercept,
        'Loss': loss,
        'Accuracy': accuracy
    }

for reg, vals in coeff_dict.items():
    print(f"{reg}: Coeff: {vals['Coefficients'][0].round(2)}, Intercept: {vals['Intercept'].round(2)}, Loss: {vals['Loss'].round(3)}, Accuracy: {vals['Accuracy'].round(3)}")

2. 正则化强度（C）：控制拟合训练数据和保持模型简单之间的权衡。C 越小，正则化越强。

# List of regularization strengths to try for L1
strengths = [0.001, 0.01, 0.1, 1, 10, 100]

coeff_dict = {}

for strength in strengths:
    lr_clf = LogisticRegression(penalty='l1', C=strength, solver='saga')
    lr_clf.fit(X_train, y_train)

    coefficients = lr_clf.coef_
    intercept = lr_clf.intercept_

    predicted_probs = lr_clf.predict_proba(X_train)[:, 1]
    loss = -np.mean(y_train * np.log(predicted_probs) + (1 - y_train) * np.log(1 - predicted_probs))
    predictions = lr_clf.predict(X_test)

    accuracy = accuracy_score(y_test, predictions)

    coeff_dict[f'L1_{strength}'] = {
        'Coefficients': coefficients[0].round(2),
        'Intercept': round(intercept[0],2),
        'Loss': round(loss,3),
        'Accuracy': round(accuracy*100,2)
    }

print(pd.DataFrame(coeff_dict).T)

# List of regularization strengths to try for L2
strengths = [0.001, 0.01, 0.1, 1, 10, 100]

coeff_dict = {}

for strength in strengths:
    lr_clf = LogisticRegression(penalty='l2', C=strength, solver='saga')
    lr_clf.fit(X_train, y_train)

    coefficients = lr_clf.coef_
    intercept = lr_clf.intercept_

    predicted_probs = lr_clf.predict_proba(X_train)[:, 1]
    loss = -np.mean(y_train * np.log(predicted_probs) + (1 - y_train) * np.log(1 - predicted_probs))
    predictions = lr_clf.predict(X_test)
    accuracy = accuracy_score(y_test, predictions)

    coeff_dict[f'L2_{strength}'] = {
        'Coefficients': coefficients[0].round(2),
        'Intercept': round(intercept[0],2),
        'Loss': round(loss,3),
        'Accuracy': round(accuracy*100,2)
    }

print(pd.DataFrame(coeff_dict).T)

3. Solver：用于优化的算法（'liblinear'、'newton-cg'、'lbfgs'、'sag'、'saga'）。某些正则化可能需要特定的算法。

4. 最大迭代次数：求解器收敛的最大迭代次数。

对于我们的高尔夫数据集，我们可能以“l2”惩罚、“liblinear”求解器和 C=1.0 作为基线。

优点和缺点

与机器学习中的任何算法一样，逻辑回归有其优点和局限性。

优点：

1. 简单：易于实现和理解。
2. 可解释性：权重直接显示每个特征的重要性。
3. 效率：不需要太多的计算能力。
4. 概率输出：提供概率而不仅仅是分类。

缺点：

1. 线性假设：假设特征和结果的对数几率之间存在线性关系。
2. 特征独立性：假设特征不是高度相关的。
3. 有限的复杂性：在决策边界高度非线性的情况下可能会出现欠拟合。
4. 需要更多数据：需要相对较大的样本量才能获得稳定的结果。

在我们的高尔夫示例中，逻辑回归可能提供一个清晰、可解释的模型，说明每个天气因素如何影响打高尔夫的决定。但是，如果决策涉及天气条件之间的复杂相互作用，而线性模型无法捕捉这些相互作用，那么逻辑回归可能很难发挥作用。

结语

逻辑回归是一种强大而又简单的分类工具。它以处理复杂数据同时又易于解释的能力而脱颖而出。与其他一些基本模型不同，它提供平滑的概率估计，并且与许多特征配合良好。在现实世界中，从预测客户行为到医疗诊断，逻辑回归通常表现得非常出色。它不仅仅是一个垫脚石——它是一种可靠的模型，在许多情况下可以匹配更复杂的模型。

逻辑回归代码总结

# Import required libraries
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

# Load the dataset
dataset_dict = {
    'Outlook': ['sunny', 'sunny', 'overcast', 'rainy', 'rainy', 'rainy', 'overcast', 'sunny', 'sunny', 'rainy', 'sunny', 'overcast', 'overcast', 'rainy', 'sunny', 'overcast', 'rainy', 'sunny', 'sunny', 'rainy', 'overcast', 'rainy', 'sunny', 'overcast', 'sunny', 'overcast', 'rainy', 'overcast'],
    'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0, 72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0, 88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
    'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0, 90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0, 65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
    'Wind': [False, True, False, False, False, True, True, False, False, False, True, True, False, True, True, False, False, True, False, True, True, False, True, False, False, True, False, False],
    'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
}
df = pd.DataFrame(dataset_dict)

# Prepare data: encode categorical variables
df = pd.get_dummies(df, columns=['Outlook'],  prefix='', prefix_sep='', dtype=int)
df['Wind'] = df['Wind'].astype(int)
df['Play'] = (df['Play'] == 'Yes').astype(int)

# Split data into training and testing sets
X, y = df.drop(columns='Play'), df['Play']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)

# Scale numerical features
scaler = StandardScaler()
float_cols = X_train.select_dtypes(include=['float64']).columns
X_train[float_cols] = scaler.fit_transform(X_train[float_cols])
X_test[float_cols] = scaler.transform(X_test[float_cols])

# Train the model
lr_clf = LogisticRegression(penalty='l2', C=1, solver='saga')
lr_clf.fit(X_train, y_train)

# Make predictions
y_pred = lr_clf.predict(X_test)

# Evaluate the model
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")

参考：

https://towardsdatascience.com/logistic-regression-explained-a-visual-guide-with-code-examples-for-beginners-81baf5871505

https://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/mlbook.html