上QQ阅读APP看书，第一时间看更新

Important evaluation metrics – regression algorithms

Assessing the value of a ML model is a two-phase process. First, the model has to be evaluated for its statistical accuracy, that is, whether the statistical hypotheses are correct, model performance is outstanding, and the performance holds true for other independent datasets. This is accomplished using several model evaluation metrics. Then, a model is evaluated to see if the results are as expected as per business requirement and the stakeholders genuinely get some insights or useful predictions out of it.

A regression model is evaluated based on the following metrics:

Mean absolute error (MAE): It is the sum of absolute values of prediction error. The prediction error is defined as the difference between predicted and actual values. This metric gives an idea about the magnitude of the error. However, we cannot judge the direction of whether the model has overpredicted or underpredicted. One should always aim for a low MAE score:

Where, y_i = Actual values

= Predicted values

n = Number of cases (records)

Mean squared error: It is the average of sum of squared errors. This metric describes both the magnitude as well as the direction of the error. However, the unit of measurement is changed as the values are squared. This deficiency is filled by another metric: root mean square error. The lower the score, the better the model is:

Root mean square error (RMSE): This metric is calculated by the square root of the mean squared error. Taking a square root converts the unit of measurement back to the original units. A model with a low RMSE score is a good model:

R² score: It is also known as coefficient of determination. It describes the percentage of variance explained by the model. For example, if R² is 0.9, then the attributes or features used in the model can represent 90% of its variation. R² varies from 0 to 1, and the higher this value, the better the model is. However, one needs to have a good testing strategy in place to validate that the model doesn't overfit:

Where, y_i = Actual values

= Predicted values

n = Number of cases (records)

= Mean of y

Overfitting occurs when a machine learning model learns the training data very well. These models have low bias and high variance in their results. In such cases, the model might lead to poor predictions on new data.

In this section, we learned about regression analysis as one of the supervised ML methods. It can be used in scenarios where the target data is continuous numerical data, such as predicting the desired salary of an employee, predicting house prices, or predicting spend values.

What if the target has discrete data? How do we predict whether a customer will churn or not? How do we predict whether a loan/credit card should be approved for a prospect? Linear regression will not work for these cases as these problems violate its underlying assumptions. Do we have any other methods? For these situations, we can use classification models.

Classification modeling is another form of supervised machine learning methods that is used to predict targets with discrete input target values. The classification algorithms are known as classifiers, as they identify the set of categories that input data support and use this information to assign a class to an unidentified or unknown target label.

In the next sections, we will walk through some of the widely-used classifiers, such as logistics regression, decision trees, SVMs, and k-Nearest Neighbors. Logistics regression can be considered as a bridge between regression and classification methods. It is a classifier camouflaged with a regression in its signature. However, it is one of the most effective and explainable classification models.