Icon

07.03 Dimensionality Reduction solution

<p>07.02 Dimensionality Reduction - solution<br><br>[L4-ML] Machine Learning Algorithms - Specialization</p><p>07 Data Cleaning Preparation<br>Apply the following dimensionality reduction techniques to the data:<br>- Filter out columns with a low variance<br>- Filter out one of two columns with a high linear correlation<br>- Replace numeric columns with principal components<br>- Filter out columns which are not important in predicting the target column</p>

URL: Ames Housing Dataset on kaggle https://www.kaggle.com/prevek18/ames-housing-dataset
URL: Description of the Ames Iowa Housing Data https://rdrr.io/cran/AmesHousing/man/ames_raw.html

07 - Data Cleaning & Preparation

07.03 Dimensionality Reduction

Learning objective: In this exercise you learn how to clean the data to build a better model, in particular by applying dimensionality reduction techniques.


Workflow description: In this workflow, Dimensionality Reduction will be implemented in both training and test datasets, using three types of techniques: Low Variance, Linear Correlation and PCA


You'll find the instructions to the exercises in the yellow annotations.

Step 1. Dimensionality Reduction by Low Variance

  • Use the Normalizer node to apply min-max normalization to the training set

  • Filter out columns in the training set that have variance lower than 0.01 (Low Variance Filter node)


Data Preparation


Step 2. Dimensionality Reduction by Linear Correlation

Filter out columns in the training set that have Linear Correlation higher or equal to 0.8 with another column (Linear Correlation and Correlation Filter nodes)


Step 4. Apply these dimensionality reduction techniques to the test set (Reference Column Filter, Normalizer (Apply) and PCA Apply nodes)

Step 3. Dimensionality Reduction by PCA

Apply automatic dimensionality reduction by replacing the numeric columns with principal components. Retain 90 % of the information in the original numeric columns. (PCA Compute and PCA Apply nodes)

  • note: PCA required z-score normalized input values

  • use Reference Column Filter node to apply the filtering of the Low Variance Filter and Correlation Filter nodes to the original input values

  • normalize using z-score (Normalizer node)


continue with unnormalized values
Reference Column Filter
Outlier Detection
Missing Value Handling
apply to the test set
Reference Column Filter
exclude columns with variance <0.01
Low Variance Filter
Fraction to preserve
PCA Apply
Compute PCAs
PCA Compute
Read AmesHousing.csv
CSV Reader
Fraction to preserve
PCA Apply
Max 0.8
Correlation Filter
Preprocessing
Calculate linear correlation
Linear Correlation
Apply to test set
Normalizer (Apply)
min-max normalization
Normalizer
z-score
Normalizer

Nodes

Extensions

Links