Icon

02_​Scaling_​Analytics_​w_​BigData

Select Options Remote executors Selected years External Data Sources: - Weather - Geo-coordinates - Radar Image Data - Storm Watch Text Data - Calendar - IBM Watson Comments - Mechanics data from web crawling
Read Data & Apply Constraints (ETL) in Hive - Local or Remote file reading (unzip) - Constraints (delay definition, selected origin airport, remove cancelled flights) - Data Blending - Dimensionality Reduction
Move from Hive to Spark 2.0 Seamlessly convert a Hive query into a Spark RDD
Pre-processing in Spark 2.0 Seamlessly convert a Hive query into a Spark RDD
Model Selection in Spark 2.0 - Pre-processing (discretization, binning, normalization, rows reduction) - Bag of Models - Decision Tree & Random Forest - Neural Network & Deep Learning - Gradient Boosted Trees - Logistic Regression (native & from Python) - Your own Ensemble Model - Current Model - Model Selection
Save the Best Model in KNIME Analytics Platform Write the model locally
Besides Hive and Spark, it is also possible to execute a KNIME node in streaming and to take advantage of the GPU computational power for some machine learning algorithms, like for example deep learning.
Big Data Analytics - Model Selection to Predict Flight Departure Delays on Hive & Spark This workflow trains a number of data analytics models on Hadoop and Spark and automatically selects the best model to predict departure delays from a selected airport. Data is the airline dataset downloadable from: http://stat-computing.org/dataexpo/2009/the-data.html. Departure delay is a delay > 15min. Default selected airport is ORD. This workflow implements data reading, data blending, ETL, guided analytics, dimensionality reduction, advanced data mining models, model selection using: Hadoop, Spark, in-memory, parallelization, grid computing, multithreading and/or in-database to speed up computationally intensive operations. Data available in knime://knime.workflow/data/1_Input
Read data from local or http://
Read Data
integration of Spark ML library
Bag of Models
define dep delay
Constraints
Convert a Hive query into Spark RDD
To Spark
Write the Model
binning partitioning normalization
Pre-processing
Select options - big data platforms - data set for integration - origin airport - years for training/ testing
Select Options
too high correlation too high % missing too low variance
Dimensionality Reduction
Add data from external data sources
Data Blending
A/B testing
Model Selection
Boosting Learner Loop Start

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: koetter
Created at: 2015-12-20
On NodePit since: 2023-02-18
Last update: 2026-11-13
Created with KNIME version: v5.2.3

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!