Icon

04 Model Building on Big Data - Solution

Solution to an L4-BD SELF-PACED COURSE exercise:
- Train a ML model in Spark
- Read the prediction results into KNIME





Exercise 4: Model Building on Big DataIn this exercise you'll train a prediction model in Spark1) Create a local big data environment. You can use the default configuration.2) Read the Spark/airline_training.parquet and Spark/airline_test.parquet folders into Spark (Parquet to Spark nodes). If the files don'texist, execute the Write parquet files metanode first.3) Filter out the following columns from the training set:- ArrTime- DepTime- All .*Delay columns but the target column DepartureDelay- UniqueCarrier, TailNum, and Origin4) Train a Random Forest model to predict departure delay- Apply entropy as the quality measure and increase the maximum depth to 105) Apply the model to the test set. Append individual class probabilities.6) Check the confusion matrix of the model7) Draw an ROC curve of the model Execute only if training and test setsdon't exist airline_training.parquetairline_test.parquetDelay columnsArrTimePredict DepartureDelay Parquet to Spark Create Local BigData Environment Parquet to Spark Spark Column Filter Spark RandomForest Learner Spark Predictor(Classification) Spark to Table ROC Curve Spark Scorer Write parquet files Exercise 4: Model Building on Big DataIn this exercise you'll train a prediction model in Spark1) Create a local big data environment. You can use the default configuration.2) Read the Spark/airline_training.parquet and Spark/airline_test.parquet folders into Spark (Parquet to Spark nodes). If the files don'texist, execute the Write parquet files metanode first.3) Filter out the following columns from the training set:- ArrTime- DepTime- All .*Delay columns but the target column DepartureDelay- UniqueCarrier, TailNum, and Origin4) Train a Random Forest model to predict departure delay- Apply entropy as the quality measure and increase the maximum depth to 105) Apply the model to the test set. Append individual class probabilities.6) Check the confusion matrix of the model7) Draw an ROC curve of the model Execute only if training and test setsdon't exist airline_training.parquetairline_test.parquetDelay columnsArrTimePredict DepartureDelay Parquet to Spark Create Local BigData Environment Parquet to Spark Spark Column Filter Spark RandomForest Learner Spark Predictor(Classification) Spark to Table ROC Curve Spark Scorer Write parquet files

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Maarit Laukkanen
Created at: 2021-03-11
On NodePit since: 2023-09-25
Last update: 2024-11-04
Created with KNIME version: v4.5.0
Tags: big dataspark

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!