Icon

07.01 Missing Value Handling exercise

<p>07.01 Missing Value Handling - exercise<br><br>[L4-ML] Machine Learning Algorithms - Specialization</p><p>07 Data Cleaning Preparation<br>Handle missing values in the data by<br>- Setting them to a fixed value (zero)<br>- Generating a dummy column based on missing values in another column<br>- Replacing them with the column mean or the most frequent value in the column<br>- Looking for different missing value patterns in the data<br>- Filtering out columns that have many missing values</p>

URL: Description of the Ames Iowa Housing Data https://rdrr.io/cran/AmesHousing/man/ames_raw.html
URL: Ames Housing Dataset on kaggle https://www.kaggle.com/prevek18/ames-housing-dataset

07 - Data Cleaning & Preparation

07.01 Missing Value Handling

Learning objective: In this exercise, you learn how to clean the data to build a better model, in particular by applying techniques to remove missing values.


Workflow description: In this workflow, missing values in both the training and test datasets will be fixed.


You'll find the instructions to the exercises in the yellow annotations.

Step 1. Create Garage column (Expression node)

The "Garage Yr Blt" column (year when the garage was built) has many missing values. This means probably that these houses don't have a garage. Therefore:

  • Create a new feature "Garage" with the value 0 if the "Garage Yr Blt" feature is missing and 1 otherwise (Expression node)


Data Preparation


Step 2. Handle missing values (Missing Value node)

The dataset contains at least two columns with many missing values: "Lot Frontage" (linear feet of street connected to the property) and "Mas Vnr Area" (masonry veneer area).

  • Set the missings values in these columns to zero (Missing Value node)

  • Moreover, replace the missing values in the "Garage Yr Blt" column with the column mean (Missing Value node)

  • Other missing values are few. Remove the remaining rows containing missing values (Missing Value node)


Step 3. Apply the above missing value handling to the test set (Expression and Missing Value (Apply) nodes)

Step 4. Filtering missing values (Optional)

  • Change "NAs" into missing values (Column Auto Type Cast node)

    • Include all columns

    • Set the missing value pattern to "NA"

  • Filter out columns that have more than 30% missing values (Missing Value Column Filter node)


Step 5. Handle remaining missing values (Missing Value node) (Optional)

Set the numeric missing values to the column mean and the string to the most frequent value


Step 6. Apply the same missing value handling to the test set (Optional)

  • Handle the as missing values in the test set (Column Auto Type Cast node)

  • Remove the discarded columns from the test set (Reference Column Filter node)


Step 7. Apply the same missing value handling to the test set (Missing Value (Apply) node) (Optional)

Read AmesHousing.csv
CSV Reader
Preprocessing

Nodes

Extensions

Links