Error Analysis

Original Source: https://www.coursera.org/specializations/deep-learning

Picking a portion of the misclassified development examples and analyzing them manually to find out what the model is failing at is called ‘error analysis’.

Let’s assume we want to classify cat images. After training a model and validating it, we pick 100 images that classifier predicted as cat but actually was not a cat. It is efficient to draw a table like below to analyze features of misclassified examples.

According to the above table, 61% of the misclassified images were blurry. So we should prioritize on having our model do better on blurry images.

Among these 100 images, there can be images that the labelr labeld as cat but actually is not a cat. In the table, there is a column named ‘incorrectly labeled’. This is where we check if the label is incorrect. In this case, only 6% was incorrectly labeld so we do not have to prioritize on correcting labels.

Machine Learning Workflow

1. Set up train/developement/test sets
2. Set up metrics
3. Build an initial system quickly
4. Train on train set, then validate on development set.
5. Use bias/variance analysis and error analysis to prioritize next steps
6. Iterate to get better model