Icon

09_​Big_​Data_​Irish_​Meter_​on_​Spark_​only

Local big data Irish meter

This workflow uses a portion of the Irish Energy Meter dataset, and presents a simple analysis based on the whitepaper "Big Data, Smart Energy, and Predictive Analytics". It is intended to highlight KNIME's Big Data and Spark functionality. The workflow creates a Local Big Data Environment, loads the meter dataset to Hive, and then transfers it into Spark. It uses a series of Spark SQL nodes to create datetime fields, and then uses Spark nodes to aggregate energy usage over these datetime fields. In the wrapped metanode, it performs PCA and k-means using Spark nodes, and does some simple visualizations of the clustered data. Finally, it writes the clustered data out to both Hive and Parquet formats.

Local_Big_Data_Irish_MeterThis workflow uses a portion of the Irish Energy Meter dataset to highlight KNIME's Big Data and Spark functionality. Compute daily, day segmentpercentagesPersist aggregate results to a Hive tablePersist aggregate resultsto HDFS in Parquet formatRead MeterData Aggregations andtime series Extract date-timeattributes Spark SQL Query PCA, K-means,Scatter Plot Spark to Hive Hive to Spark Create Local BigData Environment Load Data Spark to Parquet File Reader(Complex Format) Local_Big_Data_Irish_MeterThis workflow uses a portion of the Irish Energy Meter dataset to highlight KNIME's Big Data and Spark functionality. Compute daily, day segmentpercentagesPersist aggregate results to a Hive tablePersist aggregate resultsto HDFS in Parquet formatRead MeterData Aggregations andtime series Extract date-timeattributes Spark SQL Query PCA, K-means,Scatter Plot Spark to Hive Hive to Spark Create Local BigData Environment Load Data Spark to Parquet File Reader(Complex Format)

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: sfincher
Created at: 2018-06-22
On NodePit since: 2024-01-14
Last update: 2024-04-30
Created with KNIME version: v4.4.0
Tags: HiveSparkSpark PCASpark PivotLocal Big Data EnvironmentBig DataSpark SQLParquetIoTInternet of Things

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!