Icon

Session_​4_​Orchestration

4 workflows in this folder and its subfolders

Icon
04.0_​Reset_​DB&Big_​Data_​Environment

September 2021 · This workflow resets the database by overwriting the customers and statistics tables and sets up a local big data environment and loads the usage data into it. TAGS: Education,Data engineering,Data engineer,Best practices,big data,ELT

Icon
04.1_​ETL_​Customers

October 2021 · We access, validate, prepare, and blend the data, anonymize it, and insert it to the database table "customers". Here, we additionally export the final status of the workflow. In the case some processes fail, we notify responsible people via an automated email.

Icon
04.2_​ELT_​Usage

September 2021 · Various data are collected, e.g., session start, duration, # clicks, etc., as well as the session satisfaction score (optional) The company calculates averaged statistics for each customer, e.g., total # visits, average satisfaction, etc., and updates the "statistics" table in different locations Se…

Icon
04.3_​Orchestration

October 2021 · Once the ETL on Customers data workflow is executed successfully, i.e., customer data are accessed and anonymized, and the database table "customers" is updated, the next process - ELT on Usage data - should be triggered. In this workflow, the usage data are accessed, transformed on Spark, and aggre…