Setting Metric for Machine Learning

Original Source: https://www.coursera.org/specializations/deep-learning

Single Number Evaluation Metric

We should try many different models and choose one with the best performance on dev set.

Setting single number evaluation metric will speed up this iterating process. You can just compare this single metric and select the highest one.

For example, if you want to optimize both precision and recall, you can use F1-score.

Satisficing Metric and Optimizing Metric

You can combine multiple metrics into one with a formula like we’ve seen above, but in practice, it is hard to devise such a formula.

Instead, you can set one metric an optimizing metric and others satisficing metrics. Then, only with models that satisfy satisficing metrics, you choose a model that performs best with optimizing metric.

For example, let’s say there are two relevent metrics; ‘accuracy’ and ‘running time’. It is hard to combine two metrics into one with a formula(such formula as 2*runningtime+accuracy is nonsense). We set running time a satisficing metric; models with running time less than 1500ms will pass. Then with passed models, we choose the model with highest accuracy.

Importance of Setting Appropriate Metrics

If the metric doesn’t represent your goal, optimizing your model based on that metric is a waste of time. So spend time setting appropriate metrics. If you find out during iteration that doing well on your metric does not correspond to doing well on your application, change your metric.