Icon

02_​Spark_​Executor

17 workflows in this folder and its subfolders

Icon
01_​Spark_​MLlib_​Decision_​Tree

January 2020 · This workflow demonstrates the usage of the Spark MLlib Decision Tree Learner and Spark Predictor. It also demonstrates the conversion of categorical columns into numerical columns which is necessary since the MLlib algorithms only support numerical features and labels. The workflow makes use of the…

Icon
02_​Mass_​Learning_​Event_​Prediction_​MLlib_​to_​PMML

January 2020 · This workflow demonstrates the usage of the Spark MLlib to PMML node. Together with the Compiled Model Predictor and the JSON Input/Output node it can be used to model a so called lambda architecture which learns a machine learning model at scale on historical data offline and predicts events online…

Icon
03_​PMML_​to_​Spark_​Comprehensive_​Mode_​Learning_​Mass_​Prediction

January 2020 · This workflow demonstrates the usage of the Spark Compiled Model Predictor node which converts a given PMML model into machine code and uses the compiled model to predict vast amounts of data in parallel within Apache Spark. The workflow makes use of the Create Local Big Data Environment node to cre…

Icon
04_​Parameter_​Optimization_​in_​Spark

January 2020 · This workflow mixes standard KNIME nodes with the Spark nodes to find the optimal parameters for a k-means clustering using the hillclimbing approach. Other optimization strategies are available - check the Parameter Optimization Loop Start Node description for more. The workflow makes use of the Cr…

Icon
05_​Hive_​to_​Spark_​to_​Hive

May 2015 · This workflow demonstrates the usage of the Hive to Spark and Spark to Hive nodes that allow you to transfer data between Apache Spark and Apache Hive. To run this workflow on a remote cluster, use an HDFS Connection node, Hive Connector node, and Create Spark Context (Livy) node (available in the K…

Icon
06_​Modularized_​Spark_​Scripting

January 2020 · This workflow demonstrates the usage of the different Spark Java Snippet nodes to read a text file from HDFS, parse it, filter it and write the result back to HDFS. You might also want to have a look at the provided snippet templates that each of the node provides. In order to do so simply open the…

Icon
07_​SparkSQL_​meets_​HiveQL

January 2020 · This workflow builds a line plot of the age distribution for men and women in Maine (US) over the last 5 years. In particular, women's data is processed via Hive SQL, and men's data via Spark SQL. Will they blend? The whole data set is initially read from a Hadoop Hive installation. .... and yes, Sp…

Icon
08_​Learning_​Asociation_​Rule_​for_​Next_​Restaurant_​Prediction

June 2018 · In this workflow we demonstrate how to use the KNIME Spark nodes for giving locality recommendations. For this we are using the Yelp reviews as provided by the Kaggle challenge (https://www.kaggle.com/yelp-dataset/yelp-dataset). We wanted to find good next localities (e.g., restaurants) for everyone…

Icon
09_​Big_​Data_​Irish_​Meter_​on_​Spark_​only

January 2020 · Local big data Irish meter This workflow uses a portion of the Irish Energy Meter dataset, and presents a simple analysis based on the whitepaper "Big Data, Smart Energy, and Predictive Analytics". It is intended to highlight KNIME's Big Data and Spark functionality. The workflow creates a Local Big…