Icon

04.2_​ELT_​Usage

The company tracks the usage of the website and stores the information about each session.

- Various data are collected, e.g., session start, duration, # clicks, etc., as well as the session satisfaction score (optional)
- The company calculates averaged statistics for each customer, e.g., total # visits, average satisfaction, etc., and updates the "statistics" table in different locations
- Session satisfaction score column has missing values which need to be imputed, e.g., with machine learning predictions.

We access the usage data from Hive and personal data (anonymized & updated in sessions 1 & 2) and contracts data from a database. We perform in-database processing, read the data into Spark, enrich the usage data with the personal and contract data to predict missing values better, aggregate usage data on Spark, and save the aggregated data.

Here, we additionally export the final status of the workflow. In the case some processes fail, we notify responsible people via an automated email.

Session 4 - Orchestration

Exercise 02 ELT Usage

Transform

Learning objective: This workflow doesn't have any exercises. It already contains configured error handling and orchestration nodes. Their configuration is analogous to the configuration of the nodes in exercise 04.1_ETL_Customers. You can move to the exercise 04.3_Orchestration. There, you will trigger the execution of this workflow.


Workflow description: The company tracks the usage of the website and stores the information about each session.

  • Various data are collected, e.g., session start, duration, # clicks, etc., as well as the session satisfaction score (optional)

  • The company calculates averaged statistics for each customer, e.g., total # visits, average satisfaction, etc., and updates the "statistics" table in different locations

  • Session satisfaction score column has missing values which need to be imputed, e.g., with machine learning predictions.

We access the usage data from Hive and personal data (anonymized & updated in sessions 1 & 2) and contracts data from a database. We perform in-database processing, read the data into Spark, enrich the usage data with the personal and contract data to predict missing values better, aggregate usage data on Spark, and save the aggregated data.

Here, we additionally export the final status of the workflow. In the case some processes fail, we notify responsible people via an automated email.

Status: success
Variable Creator
Access & Transformation
Status: failure
Variable Creator
Try (Variable Ports)
Saving
Variable Creator
Workflow execution status
Workflow Service Output
Missing Value Imputation
Variable Filter
Catch Errors (Var Ports)

Nodes

Extensions

Links