9.3 Regression Evaluation
: 30 minutes
(James et al. 2021) Page 29 (mse), 69 (r2).
A regression task involves predicting a quantitative (numerical) outcome based on one or more input variables. For example, predicting someone’s income based on their age, education, and job title.
The goal is to model the relationship between predictors and a continuous response.
Y \approx f(X) + \varepsilon
Y is the quantitative response, X represents the predictor(s), f(X) is an unknown function capturing the relationship between X and Y, and \varepsilon is the irreducible error term.
Loss Functions
To evaluate the performance of a statistical learning method on a given data set, we need some way to measure how well its predictions actually match the observed data. That is, we need to quantify the extent to which the predicted response value for a given observation is close to the true response value for that observation.
Mean Squared Error (MSE)
The most commonly used performance measure in the regression setting is the Mean Squared Error (MSE).
\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{f}(x_i))^2
y_i is the true response for the i-th observation, \hat{f}(x_i) is the predicted value from the model \hat{f}, and n is the number of observations.
The MSE quantifies the average squared difference between the predicted values and the actual outcomes.
- A small \downarrow \text{MSE} indicates that predictions are close to the true values.
- A large \uparrow \text{MSE} MSE suggests substantial prediction error for at least some observations.
Mean Absolute Error (MAE)
The Mean Absolute Error (MAE) is another widely used regression performance metric:
\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{f}(x_i) \right|
y_i is the true response for the i-th observation, \hat{f}(x_i) is the predicted value, and n is the number of observations.
The MAE captures the average absolute deviation between predicted values and actual outcomes.
- A smaller \downarrow \text{MAE} implies more accurate predictions.
- A large \uparrow \text{MAE} that, on average, predictions deviate substantially from the true values.
MAE is less sensitive to outliers than MSE, making it preferable when large errors should not be overly penalized.
Metrics
To evaluate the performance of a regression model, we need a way to quantify how well its predictions align with the observed outcomes. Regression metrics provide numerical summaries of the discrepancy between the predicted and actual response values across a data set. These metrics help us assess whether the model makes accurate predictions and guide us in comparing different models or tuning their parameters.
\text{R}^2 Score
The \text{R}^2 metric provides a scale-free fit, taking values between 0 and 1.
\text{R}^2 = 1 - \frac{\text{RSS}}{\text{TSS}}
Total Sum of Squares (TSS) is the total variability in the response.
\text{TSS} = \sum_{i=1}^{n}(y_i - \bar{y})^2
Residual Sum of Squares (RSS) is the variability left unexplained by the model.
\text{RSS} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2
\text{R}^2 measures the proportion of variance in the response that is explained by the predictors.
- \text{R}^2 \approx 1: model explains most of the variance.
- \text{R}^2 \approx 0: model explains very little.
To interpret the regression metric, what counts as a “good” \text{R}^2 depends on context.
- High \uparrow \text{R}^2 is expected in physics/engineering.
- Low \downarrow \text{R}^2 is common in fields like psychology or marketing due to high inherent variability.
To connect it to correlation metric:
\text{R}^2 = r^2 \quad \text{where} \quad r = \text{Cor}(X, Y)
This highlights \text{R}^2 as a generalization of the squared correlation in the simple case.