Icon

07_​SparkSQL_​meets_​HiveQL

Spark SQL meets Hive SQL

This workflow builds a line plot of the age distribution for men and women in Maine (US) over the last 5 years.

In particular, women's data is processed via Hive SQL, and men's data via Spark SQL.

Will they blend? The whole data set is initially read from a Hadoop Hive installation.

.... and yes, Spark SQL and Hive SQL do blend!

Spark SQL meets Hive SQL.This workflow builds a line plot of the age distribution for men and women in Maine (US) over the last 5 years using both Spark SQL and KNIME DB nodes. Hive inDB Data Manipulation - On Female Records - Remove PWGTP* & PUMA* columns - Count number of records by AGEP Spark inDB Data Manipulation - On Male Records - Remove PWGTP* & PUMA* columns - Count number of records by AGEP SELECT * FROM #table# WHERE SEX = 1 (male)... and into KNIMECOUNT(*) FROM #table# BY AGEPfilling ageholesline plotrm PUMA*& PWGTP*convert a Hive queryinto a Spark RDDselect * fromss13pme tableonly Femalerecordsrm PUMA*& PWGTP*count recordsBY AGEP... and into KNIMEblend data Spark SQL Query Spark to Table Spark SQL Query Fix Missing Values Visualization Spark SQL Query Hive to Spark DB Table Selector Read Data IntoLocal Spark Env DB Row Filter DB Column Filter DB GroupBy DB Reader Joiner Spark SQL meets Hive SQL.This workflow builds a line plot of the age distribution for men and women in Maine (US) over the last 5 years using both Spark SQL and KNIME DB nodes. Hive inDB Data Manipulation - On Female Records - Remove PWGTP* & PUMA* columns - Count number of records by AGEP Spark inDB Data Manipulation - On Male Records - Remove PWGTP* & PUMA* columns - Count number of records by AGEP SELECT * FROM #table# WHERE SEX = 1 (male)... and into KNIMECOUNT(*) FROM #table# BY AGEPfilling ageholesline plotrm PUMA*& PWGTP*convert a Hive queryinto a Spark RDDselect * fromss13pme tableonly Femalerecordsrm PUMA*& PWGTP*count recordsBY AGEP... and into KNIMEblend dataSpark SQL Query Spark to Table Spark SQL Query Fix Missing Values Visualization Spark SQL Query Hive to Spark DB Table Selector Read Data IntoLocal Spark Env DB Row Filter DB Column Filter DB GroupBy DB Reader Joiner

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: koetter
Created at: 2015-07-31
On NodePit since: 2019-07-12
Last update: 2024-04-18
Created with KNIME version: v4.7.0
Tags: SparkHiveHadoopBig DataSQLin-databasewill they blend

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!