Icon

ML Pipeline with Spark Streaming

DATA ACCESS MODEL TRAINING & PREDICTION POSTPROCESSING OUTPUT DATA DATA PREPROCESSING & PARTITIONING UPDATE DATA OUTPUT BACK TO DATABASE DASHBOARD ML data pipeline and training VIA SPARK STREAMING APPLICATIONThis workflow demo on data streaming via spark from database to ML model training. The output is then post-processed and updated back to database via sparks streaming application.For visualization, the output table is send to Power BI for a nice dashboard visualisation. create local big dataenv (prerequisite: require hadoopinstallation)send data from DB to sparkselect schema& tablestart connectionto MySqlvisualizeActual Vs Predictedupdate table in DBvisualizeActual Vs Predictedmodel evaluationmodel evaluationsend table to sparksend data to power BIprovide Microsoftloginjoin both output tablefilter andrename selected column Create Local BigData Environment Persist SparkDataFrame/RDD DB to Spark DB Table Selector MySQL Connector Spark to Table Line Plot Spark to DB Random Forest(Regression) Gradient BoostedTree (Regression) Input dataPreprocessing Data splittingand Normalizing Line Plot Numeric Scorer Numeric Scorer Metanode Metanode Table to Spark Send to Power BI MicrosoftAuthentication DB Reader Joiner Table Manipulator DATA ACCESS MODEL TRAINING & PREDICTION POSTPROCESSING OUTPUT DATA DATA PREPROCESSING & PARTITIONING UPDATE DATA OUTPUT BACK TO DATABASE DASHBOARD ML data pipeline and training VIA SPARK STREAMING APPLICATIONThis workflow demo on data streaming via spark from database to ML model training. The output is then post-processed and updated back to database via sparks streaming application.For visualization, the output table is send to Power BI for a nice dashboard visualisation. create local big dataenv (prerequisite: require hadoopinstallation)send data from DB to sparkselect schema& tablestart connectionto MySqlvisualizeActual Vs Predictedupdate table in DBvisualizeActual Vs Predictedmodel evaluationmodel evaluationsend table to sparksend data to power BIprovide Microsoftloginjoin both output tablefilter andrename selected column Create Local BigData Environment Persist SparkDataFrame/RDD DB to Spark DB Table Selector MySQL Connector Spark to Table Line Plot Spark to DB Random Forest(Regression) Gradient BoostedTree (Regression) Input dataPreprocessing Data splittingand Normalizing Line Plot Numeric Scorer Numeric Scorer Metanode Metanode Table to Spark Send to Power BI MicrosoftAuthentication DB Reader Joiner Table Manipulator

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: ROG
Created at: 2022-09-02
On NodePit since: 2022-09-16
Last update: 2024-04-28
Created with KNIME version: v4.6.1

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!