Setting Development Set and Test Set

Original Source: https://www.coursera.org/specializations/deep-learning

Development set is also called hold-out validation set, validation set, or dev set.

Size of the Development and Test Sets

Set your dev set to be big enough to detect differences in algorithm/models you’re trying out. Set your test set to be big enough to give high confidence in the overall performance of your system.

You don’t need to follow tradition of splitting dataset into 6:2:2. If you have large dataset, you might only need 100000 development examples.

Distribution of Dev and Test Sets.

You need to have dev set and test set come out from the same distribution.

For example, suppose you want to sort out cat images from images that users upload on your app. You trained and validated on the high-resolution images you’ve found on internet. However, images users actually upload is low-resolution. Your model that have been optimized to high-resolution images might not do well on low-resolution images. So you should validate with low-resolution images in this case.

If the dataset is small, you should prioritize on making dev set come from the same distribution as real world data, rather than train set.