Icon

04 Model Building on Big Data

04 Model Building on Big Data
Execute only if training and test setsdon't exist in HDFS Exercise 4: Model Building on Big DataIn this exercise you'll train a prediction model in Spark1) Create a local big data environment. You can use the default configuration.2) Read the Spark/airline_training.parquet and Spark/airline_test.parquet folders into Spark (Parquet to Spark nodes). If the files don'texist, execute the Write parquet files metanode first.3) Filter out the following columns from the training set:- ArrTime- DepTime- All .*Delay columns but the target column DepartureDelay- UniqueCarrier, TailNum, and Origin4) Train a Random Forest model to predict departure delay- Apply entropy as the quality measure and increase the maximum depth to 105) Apply the model to the test set. Append individual class probabilities.6) Check the confusion matrix of the model7) Draw an ROC curve of the model Write parquet files Execute only if training and test setsdon't exist in HDFS Exercise 4: Model Building on Big DataIn this exercise you'll train a prediction model in Spark1) Create a local big data environment. You can use the default configuration.2) Read the Spark/airline_training.parquet and Spark/airline_test.parquet folders into Spark (Parquet to Spark nodes). If the files don'texist, execute the Write parquet files metanode first.3) Filter out the following columns from the training set:- ArrTime- DepTime- All .*Delay columns but the target column DepartureDelay- UniqueCarrier, TailNum, and Origin4) Train a Random Forest model to predict departure delay- Apply entropy as the quality measure and increase the maximum depth to 105) Apply the model to the test set. Append individual class probabilities.6) Check the confusion matrix of the model7) Draw an ROC curve of the model Write parquet files

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Maarit Laukkanen
Created at: 2021-03-11
On NodePit since: 2022-02-11
Last update: 2024-11-04
Created with KNIME version: v4.5.0
Tags: big dataspark

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!