In the last three weeks, we have talked about data exploring, data pre-processing and data analysis. In the data exploring stage, we use approaches such as visual exploration or statistics to understand what is in a dataset and the characteristics of the data. These characteristics can include size or amount of data, completeness of the data (e.g., the number of missing values), correctness of the data (e.g., outliers), possible relationship amongst data elements or variables in the data (e.g., correlations, distribution).

Based on the information obtained through data exploring, data pre-processing aims to improve the quality of data by dealing with missing values, removing unusable parts of the data, correcting poorly formatted elements and defining relevant relationships across datasets. Common approaches for data pre-processing include missing value imputation, data scaling, normalization, Feature selection and Dimension reduction.

After data exploring and data pre-processing, data are prepared for analysis. Machine learning algorithms are widely used in data analysis. There are various machine learning algorithms, which can be grouped into different types. Supervised learning and unsupervised learning are two major types of machine learning. In this module, we introduced three supervised machine learning algorithms, i.e., Linear regression, SVM (Support Vector Machine) and NN (Neural Network), and one unsupervised machine learning algorithm: K-means Clustering.

Training, Test, and Validation Datasets

Ion machine learning, multiple datasets are used to build final model. In particular, three datasets are commonly used in different stage of the construction of the model, which are training dataset, test dataset and validation dataset.

Definitions

Training dataset
Training dataset is the actual dataset used to train the model. The model learns from this data, uses the data to fit the parameters (e.g., the weights and bias in the Neural Network).
Generally, training data is a certain percentage of an overall dataset along with testing set. As a rule, the better the training data, the better the algorithm or classifier performs.

Validation dataset
Validation dataset is used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyper-parameters. The model occasionally ‘sees’ this dataset for frequent evaluation, but never learns from this dataset.

Test dataset
Test dataset is used to provide an unbiased evaluation of a final model fit on the training dataset.

During training only the training and/or validation set is available. The test dataset must not be used during training. The test dataset will only be available during testing the classifier. Generally, the term “validation dataset” is used interchangeably with the term “test dataset” and refers to a sample of the dataset held back from training the model.

Often times, these data sets are taken from the same overall dataset. By using similar data for training and testing, you can minimize the effects of data discrepancies and better understand the characteristics of the model.

Split up dataset into test and training sets

Holdout method
Most simply, the original dataset is split up more or less randomly into training and test dataset, while making sure the training dataset to capture important classes you know up front.
Typically, when you separate a data set into a training set and a testing set, most of the data (e.g. 80%) is used for training, and a smaller portion of the data (e.g. 20%) is used for testing.
Holdout method is widely used for the test-training dataset specification. However, doing so can bias the classification results and the results may not be generalizable.

N-Fold cross validation
This approach randomizes the dataset and create N (almost) equal size partitions, and chooses the Nth partition for testing and the other N-1 partitions for training (within the training set you can further employ another K-fold cross validation to create a validation set and find the best parameters). Repeat this process N times to get an average of the metric.
Cross-validation is almost unbiased. However, cross-validation does not work in situations where the data cannot be shuffled, most notably in time-series.