Icon

01_​Big_​Data_​Preprocessing_​Example

Big Data preprocessing

This workflow demonstrates the usage of the DB nodes in conjunction with the Create Local Big Data Environment node, which is part of the KNIME Big Data Extension. This node, together with the DB nodes, allows complex data preprocessing without the need of manual SQL coding.

To run this workflow on a remote cluster, use an HDFS Connection node and Hive Connector node (available in the KNIME Big Data Connectors Extension) in place of the Create Local Big Data Environment node.

The table name is controlled by a workflow variable which can be altered via the context menu of the workflow in the KNIME explorer.

Requirements:
- KNIME File Handling Nodes
- KNIME Extension for Local Big Data Environments

Create Table
Big Data Preprocessing Please run the branches below in order for best results: Create Table --> Process Data --> Drop Table. For more information see the workflow metadata. Find it here: View -> Description
Process Data
Drop Table
DB Reader
DB Reader
set execution engine to tez
DB SQL Executor
sort by universe_0_0 ascending
DB Sorter
Sample Data
Data Generator
DB Reader
keep only Iris-versicolor
DB Row Filter
create the demo table
DB Table Creator
calculate statistics
DB GroupBy
load the demo table
DB Loader
drop test table (execute last)
DB Table Remover
select the created table
DB Table Selector
Create Local Big Data Environment
remove cluster membership
DB Column Filter
join original data with aggregated data
DB Joiner

Nodes

Extensions

Links