0 ×

09_​Big_​Data_​Irish_​Meter_​on_​Spark_​only

Workflow

Local big data Irish meter

This workflow uses a portion of the Irish Energy Meter dataset, and presents a simple analysis based on the whitepaper "Big Data, Smart Energy, and Predictive Analytics". It is intended to highlight KNIME's Big Data and Spark functionality. The workflow creates a Local Big Data Environment, loads the meter dataset to Hive, and then transfers it into Spark. It uses a series of Spark SQL nodes to create datetime fields, and then uses Spark nodes to aggregate energy usage over these datetime fields. In the wrapped metanode, it performs PCA and k-means using Spark nodes, and does some simple visualizations of the clustered data. Finally, it writes the clustered data out to both Hive and Parquet formats.

HiveSparkSpark PCASpark PivotLocal Big Data EnvironmentBig DataSpark SQLParquetIoTInternet of Things
Local_Big_Data_Irish_MeterThis workflow uses a portion of the Irish Energy Meter dataset to highlight KNIME's Big Data and Spark functionality. Read MeterDataPersist aggregate resultsto HDFS in Parquet formatCompute daily, day segmentpercentagesPersist aggregate results to a Hive table File Reader Aggregations andtime series Extract date-timeattributes Spark to Parquet Spark SQL Query PCA, K-means,Scatter Plot Create Local BigData Environment Spark to Hive Hive to Spark Load Data Local_Big_Data_Irish_MeterThis workflow uses a portion of the Irish Energy Meter dataset to highlight KNIME's Big Data and Spark functionality. Read MeterDataPersist aggregate resultsto HDFS in Parquet formatCompute daily, day segmentpercentagesPersist aggregate results to a Hive table File Reader Aggregations andtime series Extract date-timeattributes Spark to Parquet Spark SQL Query PCA, K-means,Scatter Plot Create Local BigData Environment Spark to Hive Hive to Spark Load Data

Download

Get this workflow from the following link: Download

Nodes

09_​Big_​Data_​Irish_​Meter_​on_​Spark_​only consists of the following 62 nodes(s):

Plugins

09_​Big_​Data_​Irish_​Meter_​on_​Spark_​only contains nodes provided by the following 4 plugin(s):