Data Standardization And Pipelines

In this section, we motivate and introduce additional data pre-processing techniques that involve transforming the input dataset before fitting our model. We also introduce the Pipeline class from SciKit-Learn for chaining multiple transformations together. By the end of this section, you should be able to:

Apply data pre-processing techniques that tranform the dataset such as standardization, maxabs Scaler and robust Scaler.
Identify when to use each technique.
Implement the SciKit-Learn Pipeline abstraction to conduct multi-step data analysis.

Data Pre-Processing: Motivation

Most of the data analysis methods from earlier in this workshop assumed that the variables were on a similar numeric scale to one another. Moreover, the algorithms for fitting the models we have looked at mostly work by minimizing a cost function using an algorithm like gradient descent. The partial derivatives appearing in the gradient computation are sensitive to the (changes of) actual values of the independent variables. In practice, several issues can arise:

Variables on vastly different scales make exploratory data analysis more difficult; for example, tools such as heatmap and visualizations like plots become harder to use.
One variable can have much larger values than the others and it can dominate the cost function, in which case the other variables wouldn’t contribute to the fitting as much.
Datasets containing continuous variables with large values and/or variance can make the convergence of optimization algorithms take much longer.

The idea, at a high-level, is to transform the column variables to put them on the same numeric scale; for example, between 0 and 1; -1 and 1 or between \(min\) and \(max\) for two constants, \(min, max\). However, care is needed for the following reasons:

Not every pre-processing method is applicable to every variable/dataset. For example, variables with a small number of very significant outliers can be skewed with techniques that use averages while the structure of sparse variables would typically be lost if one attempted to center it at, say 0. More generally, the techniques we will look at in this module apply to continuous variables. Categorical variables are often treated with different methods, such as 1-hot encoding, which we have discussed previously.
The parameters of a pre-processing step should be computed on only the training data (i.e., after performing the train-test split) so as to not “leak” information from the test set. However, it is very important to apply the pre-processing to the test set before predicting; otherwise, the model performance will suffer.

In addition to improving the overall modularity and reuse of our code, the Pipeline class will help in particular with point 2) above, as it will ensure the same pre-processing is applied to the test data before predict is called.

Data Standardization: Mean Removal and Variance Scaling

Data standardization, sometimes also referred to as z-Score normalization, is the process of transforming a continuous variable to have a mean of zero and a standard deviation of 1. Mathematically, the procedure is straight-forward: for each continuous feature \(X_i\) in the dataset, and each \(x \in X_i\) we make the following update:

\[x \rightarrowtail (x - mean(X_i)) / std(X_i)\]

where:

\(mean(X_i)\) is the mean of the column, \(X_i\)
\(std(X_i)\) is the standard deviation of the column, \(X_i\)

It’s clear that the updated values in the \(X_i\) column have mean 0 and standard deviation 1.

Applying data standardization to continuous columns in a dataset can be an important pre-processing step when the column variables have a normal distribution – for example, not sparse, and no significant outliers.

When to Use: When the dataset is normally distributed (or close to it).

StandardScaler in SciKit-Learn

The StandardScaler class from the sklearn.preprocessing module provides a convenience class that implements data standardization. The classes in preprocessing module that perform transformations on data are all used in the following way:

Instantiate an instance of the class
Fit the instance to the training data using the .fit() function.
Apply the transformation to a dataset using the .transform() function.

Note that we always apply the fit() to the training data to “learn” the scaling parameters (in this case the mean and standard deviation). We never apply it to test data, as this would cause our model to be fit in part based on the test data.

Warning

Using test data to fit the Scaler can lead to overly optimistic performance estimates. A simple rule to remember is this: Never call fit() on the test data.

Let’s see an example using our iris data set:

>>> from sklearn import datasets
>>> import numpy as np
>>> iris = datasets.load_iris()

>>> from sklearn.model_selection import train_test_split
>>> X = iris.data
>>> y = iris.target
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)

>>> # reminder of what the columns contain
>>> print(iris.feature_names)
>>> print(X_train[:10])
['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
array([[7.7, 2.6, 6.9, 2.3],
       [5.7, 3.8, 1.7, 0.3],
       [5. , 3.6, 1.4, 0.2],
       [4.8, 3. , 1.4, 0.3],
       [5.2, 2.7, 3.9, 1.4],
       [5.1, 3.4, 1.5, 0.2],
       [5.5, 3.5, 1.3, 0.2],
       [7.7, 3.8, 6.7, 2.2],
       [6.9, 3.1, 5.4, 2.1],
       [7.3, 2.9, 6.3, 1.8]])

>>> # print the column means and standard deviations
>>> print(f'mean by column:   {np.mean(X_train, axis=0)}')
>>> print(f'stddev by column: {np.std(X_train, axis=0)}')
mean by column:   [5.8        3.03809524 3.73809524 1.19047619]
stddev by column: [0.84052138 0.4168775  1.78012204 0.77931885]

We see that prior to standardization the columns each have a positive mean and non-normal standard deviation. Next, transform the data using the standard Scaler:

>>> from sklearn.preprocessing import StandardScaler

>>> # step 1 -- Instantiate the Scaler
>>> iris_Scaler = StandardScaler()
>>> # step 2 -- fit the Scaler to the training data
>>> iris_Scaler.fit(X_train)
>>> # step 3 -- apply the transformation; in this case, we apply it to the training data.
>>> X_train_scaled = iris_Scaler.transform(X_train)

>>> # print the column means and standard deviations after transformation
>>> print(f'scaled mean by column:   {np.mean(X_train_scaled, axis=0)}')
>>> print(f'scaled stddev by column: {np.std(X_train_scaled, axis=0)}')
scaled mean by column:   [-8.47998920e-16 -1.81442163e-15  2.11471052e-17 -5.96348368e-16]
scaled stddev by column: [1. 1. 1. 1.]

We see that the mean of the dataset after applying the transformation is (essentially) 0 and the standard deviation is 1.

Note

Even though the above method works fine, we recommend using the Pipeline class described at the end of this section when combining data preprocessing with model training.

Robust Scalers

When the dataset contains outliers that deviate significantly from the mean, using standardization could result in worse performance because the outliers could dominate the mean/variance and crush the signal.

In these cases, a robust Scaler based on different statistical methods, such as IQR, can be used instead. With a robust Scaler, the median is removed, and scaling is performed based on some percentage range.

When to Use: When the dataset contains outliers that deviate significantly from the mean.

RobustScaler in SciKit-Learn

The RobustScaler class in SciKit-Learn provides the same methods as the StandardScaler we just looked at. Just like before, we’ll follow the following steps:

Instantiate an instance of the class
Fit the instance to the training data using the .fit() function.
Apply the transformation to a dataset using the .transform() function.

We’ll look at an example of RobustScaler in the section on Pipelines. For now, let’s take a quick example involving a plain numpy array.

>>> # define a numpy array with an outlier --- most of the values are
>>> # around 10, but there is one value of 10,000,000:
>>> n = np.array([10, 11, 9, 8, 8.5, 10000000, 9, 10, 10])

>>> print(n.mean(), np.median(n), n.std())
1111119.5 10 3142693.8393535535

We see the that the mean and standard deviation are large, while the median is 10. Let’s try scaling this array using both StandardScaler and RobustScaler. Note that we have to reshape the array to instruct the scaler that it should be treated as a single column feature (if it were a single sample consisting of multiple columns, we should reshape is with reshape(1, -1)).

>>> from sklearn.preprocessing import RobustScaler, StandardScaler
>>> # 30 normally distributed points with mean 5 and std 3
>>> data = np.random.normal(5, 3, 20)
>>> df1 = pd.DataFrame({"data": data})
>>> print(df1.describe())

>>> # some outliers
>>> outliers = np.array([150, 600, 900])
>>> df2 = pd.DataFrame({
>>>     "data2": np.append(data, outliers)
>>> })
>>> print(df2.describe())

            data2
count   23.000000
mean    75.203711
std    219.806640
min     -4.457382
25%      2.587355
50%      5.318264
75%      6.964271
max    900.000000

Now, let’s apply a robust scaler:

>>> robust_scaler = RobustScaler().fit(df2)
>>> robust_scaled_data = robust_scaler.transform(df2)

Let’s see what these scalers did to the data:

>>> robust_scaled_df = pd.DataFrame({"data": robust_scaled_data.reshape(-1)})
>>> robust_scaled_data.describe()

            data
count   23.000000
mean    15.966825
std     50.219529
min     -2.233456
25%     -0.623935
50%      0.000000
75%      0.376065
max    204.409182

Discussion: Note that the range of values is still quite wide after applying the robust scaler. By comparison, what do you think would happen if we applied the StandardScaler to these data?

The range would be much more narrow.

MaxAbs Scaler

The last Scaler we will mention is the MaxAbsScaler, short for “maximum absolute” scaler. This scaler uses the maximum absolute value of each feature to scale the values of that feature (i.e., the maximum absolute values of each feature after transformation will be 1). Note that it does not attempt to shift/center the data, so if a feature is sparse (i.e., consists mostly of 0s), the data “spareness” structure will not be destroyed.

Note also that this scaler does not reduce the effect of outliers.

When to Use: When the dataset contains sparse data.

Pipelines

The SciKit-Learn package provides a utility class called Pipeline that can be used to make your code more modular/reusable and to ensure that the same preprocessing steps are applied to training and test data in the appropriate way.

The idea of the Pipeline is to define a sequence of transformations to preprocess data and fit the model. The intermediate steps can be any transformation that implement the Transforms API.

There are a couple of ways of constructing Pipeline objects. The first way we will look at is with the make_pipeline() convenience function from the sklearn.pipeline module. This method is good for simple pipelines where we don’t need to refer to the attributes on objects within steps. Next, we will look at calling the Pipeline() constructor (from the same module) directly. We will need to do this when we want to combine pipelines with GridSearchCV, for example.

An Initial Pipeline

Let’s first build a pipeline to apply a scaler to the Pima Indians Diabetes dataset before fitting a KNN classifier model. In this first approach, we will hard code the number of neighbors, but we will see that the scaler already improves the performance.

To begin, we will perform some initial data load and pre-processing. For backaround on this dataset in the pre-processing steps we took, see the KNN materials.

>>> data = pd.read_csv("../Diabetes-Pima/diabetes.csv")
>>> # Glucose, BMI, Insulin, Skin Thickness, Blood Pressure contains values which are 0
>>> data.loc[data.Glucose == 0, 'Glucose'] = data.Glucose.median()
>>> data.loc[data.BMI == 0, 'BMI'] = data.BMI.median()
>>> data.loc[data.Insulin == 0, 'Insulin'] = data.Insulin.median()
>>> data.loc[data.SkinThickness == 0, 'SkinThickness'] = data.SkinThickness.median()
>>> data.loc[data.BloodPressure == 0, 'BloodPressure'] = data.BloodPressure.median()

>>> # x are the dependent variables and y is the target variable
>>> X = data.drop('Outcome',axis=1)
>>> y = data['Outcome']

>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)

Recall from the notes that we found the optimal n_neighbors to be 13 using GridSearchCV in our previous lecture. We’ll hard code the 13 value for now, but note that because we’ll be using scaling, the optimal n_neighbors value could be different.

To create a pipeline using the make_pipeline function, all we have to do is pass the objects (transformations) we want to perform as arguments in the order they should be performed. The last step of a pipeline should be the model to be fit.

Here we create a pipeline with two steps: the StandardScaler and the KNeighborsClassifier:

>>> from sklearn.pipeline import make_pipeline
>>> pipe_line = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=13))

With the pipe_line object created, we now call fit() to execute each transformation in the pipeline. We pass the train dataset, just as we would when calling fit() on the transformation or model directly:

>>> pipe_line.fit(X_train, y_train)

Finally, we call score() or a similar method to assess the model’s performance. Note that the pipeline applies all of the transformations to the test data. This ensures we get optimal model performance. If we applied a scaling method to train the model but did not apply the same method to the test data, we wold likely get poor results.

>>> print(pipe_line.score(X_test, y_test))  # apply scaling on testing data, without leaking training data.
0.7532467532467533

Note that the score function uses accuracy by default here. Our model achieves 75% accuracy on the test data. That’s already an improvement over the model we learned without scaling (recall that we had achieved 71% previously).

Note also that the other methods are available, such as predict(), on our pipe_line object, so we can do things like:

>>> from sklearn.metrics import classification_report
>>> print(classification_report(y_test, pipe.predict(X_test)))

Pipeline with Named Steps and GridSearchCV

We already saw some improvements with the simple pipeline above, but we can do better. We can search for the optimal hyperparameters (in our case, the n_neighbors) given that the dataset has been scaled.

To do that, we need to use the Pipeline constructor to name the steps of our pipeline. All we do is provide an additional argument, a string which is used for the name:

>>> from sklearn.pipeline import Pipeline

>>> p = pipeline = Pipeline([
>>>     ('scale', StandardScaler()),
>>>     ('knn', KNeighborsClassifier()),
>>> ])

Here we have defined a pipeline with two steps, just as before. We named the first step “scale” and the second one “knn”. Note that we do not specify the n_neighbors value to the KNeighborsClassifier() constructor – we’re going to search for that.

Now, we need to define our parameter grid, like we have done before, to describe the space of the parameters we want to search on. The key here is that we need to namespace the parameter by the step name, because a given parameter will only apply to a certain step.

The way to do that is to use the step name, then two underscores (i.e., __) and then the parameter name; i.e., <step_name>__<param_name>. For example, knn__n_neighbors refers to the n_neighbors attribute of the knn step. We then supply the range of values for the parameter just as before.

Here is our param_grid definition:

>>> param_grid = {
>>>     "knn__n_neighbors": np.arange(1, 100)
>>> }

With that, we can define the GridSearchCV object as before but this time passing the pipeline object instead of the model. We then call fit() and score() etc., using the search object:

>>> search = GridSearchCV(p, param_grid, n_jobs=4)
>>> search.fit(X_train, y_train)
>>> print(f"Score with best parameters: {search.best_score_}")
>>> print(search.best_params_)

Score with best parameters: 0.7820872274143303
{'knn__n_neighbors': 19}

Note that the optimal n_neighbors was 19, different from the optimal value of 13 we found without the scaling, and the accuracy has increased to 78%.

Pipeline With A Custom SciKit-Learn Model to Search Across Models

In this section, we provide an example of writing a custom model in SciKit-Learn. The idea is to allow us to search across models and hypyerparemeters within a single pipeline object. It also allows us to illustrate how relatively simple it is to extend the BaseEstimator class with custom behaviors. For more details, see the SciKit-Learn docs.

We’ll create a child class of the BaseEstimator class that accepts a model object as a parameter to the constructor and provides implementations of the fit(), predict(), predict_proba() and score() methods that utilize the model. In this way, we will be able to pass the model object as a parameter in our param_grid attribute that will be used in the pipeline and search.

Here is the code for our class:

>>> from sklearn.base import BaseEstimator
>>> from sklearn.neighbors import KNeighborsClassifier

>>> class MultiModelClassifier(BaseEstimator):
>>>     """
>>>     A custom Estimator class that can be constructed with different model types.
>>>     For details on implementing custom Estimators,
>>>     see: https://scikit-learn.org/stable/developers/develop.html
>>>     """

>>>     def __init__(self, model=KNeighborsClassifier()):
>>>         """
>>>         A custom estimator parameterized by the model.
>>>         Pass the result of an estimator constructor for `model`. By default,
>>>         it uses the KNeighborsClassifier().
>>>         """
>>>         self.model = model

>>>     def fit(self, X, y=None, **kwargs):
>>>         self.model.fit(X, y)
>>>         return self
>>>
>>>     def predict(self, X, y=None):
>>>         return self.model.predict(X)
>>>
>>>     def predict_proba(self, X):
>>>         return self.model.predict_proba(X)
>>>
>>>     def score(self, X, y):
>>>         return self.model.score(X, y)

You will see that the code is pretty straight-forward: in the constructor, all we do is save the model object that the user passed us as self.model. Then, in each of the other methods, we simply call the corresponding method on self.model.

Let’s see how to use this in a pipeline and grid search. First we define out pipeline. It will have two steps, the first one being the scaler and the second one the model. We’ll use our new MultiModelClassifier as the model step.

>>> p2 = Pipeline([
>>>     ('scale', StandardScaler()),
>>>     ('mmc', MultiModelClassifier()),
>>> ])

Now to define our parameter grid. This time, the param_grid object will be a list of dictionaries, with each dictionary corresponding to a parameter space to search over for a specific model.

We define the model to use by setting the model parameter to the mmc step using the __ notation. That is, "mmc__model" will be a key in our dictionary and will have a value which will be the model we want to use (but as a list – all the keys should be lists).

Then, we can define the associated hyperparameters to search over for that model. Keep in mind that we will need two __ since we will be referecing an attribute of the model object within the mmc step. For example, we can put mmc__model__n_neighbors to refer to the n_neighbors hyperparameter of the mmc__model object when the model is KNeighborsClassifier. Here’s a complete examples:

>>> param_grid = [
>>>     {
>>>         "mmc__model": [KNeighborsClassifier()],
>>>         "mmc__model__n_neighbors": np.arange(1, 100)
>>>     },
>>>     {
>>>         "mmc__model": [RandomForestClassifier()],
>>>         "mmc__model__n_estimators": np.arange(start=20, stop=150, step=3),
>>>     },
>>> ]

We can now construct the search object, fit and score, as before:

>>> gscv2 = GridSearchCV(p2, param_grid, cv=5)
>>> gscv2.fit(X_train, y_train)
>>> print("scaling best params: ", gscv2.best_params_)
>>> accuracy_test2 = accuracy_score(y_test, gscv2.best_estimator_.predict(X_test))
>>> print(f'Accuracy of best estimator WITH SCALING on test data is: {accuracy_test}')

scaling best params:  {'mmc__model': RandomForestClassifier(), 'mmc__model__n_estimators': 62}
Accuracy of best estimator WITH SCALING on test data is: 0.7359307359307359

The output indicates that the search found the RandomForestClassifier with 62 trees to perform best.