Regression Evaluation Metrics

When working with regression models, it's crucial to evaluate their performance by comparing the predicted and actual values. Here are some common metrics used for this purpose:

Mean Absolute Error (MAE)

The Mean Absolute Error (MAE) indicates by how much each prediction was wrong on average, without considering whether the prediction was over or under the actual value. This metric is the mean of the absolute errors for the entire validation set.

For example, consider predicting the number of ice creams sold. If the absolute errors are 2, 3, 3, 1, 2, and 3, the MAE is calculated as:

MAE = (2 + 3 + 3 + 1 + 2 + 3) / 6 = 2.33

Mean Squared Error (MSE)

While MAE treats all discrepancies equally, sometimes it's desirable to amplify larger errors. The Mean Squared Error (MSE) does this by squaring the errors before averaging them, which emphasizes larger errors more.

Using the same ice cream example, the squared errors are 4, 9, 9, 1, 4, and 9, so the MSE is:

MSE = (4 + 9 + 9 + 1 + 4 + 9) / 6 = 6

Root Mean Squared Error (RMSE)

The MSE metric is useful, but it doesn't represent the quantity measured by the label due to the squaring process. To bring it back to the original scale, we calculate the Root Mean Squared Error (RMSE), which is the square root of MSE.

For the example:

RMSE = √6 ≈ 2.45 (ice creams)

Coefficient of Determination (R²)

The Coefficient of Determination, or R², measures the proportion of variance in the actual values explained by the model. An R² value closer to 1 indicates a better fit.

The formula for R² is:

R² = 1 - ∑(y - ŷ)² / ∑(y - ȳ)²

In simple terms, if our ice cream regression model has an R² of 0.95, it means that 95% of the variance in ice cream sales can be explained by the model.

Iterative Training

In practice, developing a regression model involves iterative training. This includes:

• Feature selection and preparation: Choosing the right features and applying transformations to ensure a better model fit.
• Algorithm selection: While linear regression is one option, many other regression algorithms could be explored.
• Algorithm parameters: Adjusting hyperparameters to control the algorithm's behaviour and improve performance.

Through multiple iterations, a data scientist refines the model to achieve the best evaluation metrics suitable for the scenario.