03.2_​Missing_​value_​imputation_​on_​Spark

0 ×

03.2_Missing_value_imputation_on_Spark_solution

The company tracks the usage of the website and stores the information about each session.
- Various data are collected, e.g., session start, duration, # clicks, etc., as well as the session satisfaction score (optional)
- The company calculates averaged statistics for each customer, e.g., total # visits, average satisfaction, etc., and updates the "statistics" table on the database
- Session satisfaction score column has missing values which need to be imputed, e.g., with machine learning predictions.

We access the usage data from Hive and personal data (anonymized & updated in sessions 1 & 2) and contracts data from the PostgreSQL database. We perform in-database processing, read the data into Spark, enrich the usage data with the personal and contract data to predict missing values better, and continue working with the relatively big usage data on Spark. We export the final status of the workflow. In the case some processes fail, we notify responsible people via an automated email.

URL: L4-DE Course Slides https://www.knime.com/form/material-download-registration

Nodes

• Component Input3 ×
• Component Output3 ×
• DB Table Selector (Labs)3 ×
• String Configuration3 ×
• Spark Decision Tree Learner2 ×
• Show all 32 nodes

Extensions

• FeatureKNIME Base nodes
• FeatureKNIME Database
• FeatureKNIME Extension for Apache Spark
• FeatureKNIME Extension for Apache Spark (legacy)
• FeatureKNIME Optimization extension
• Show all 6 modules

Links

• No links available