Icon

04.3_​Orchestration

Once the ETL on Customers data workflow is executed successfully, i.e., customer data are accessed and anonymized, and the database table "customers" is updated, the next process - ELT on Usage data - should be triggered. In this workflow, the usage data are accessed, transformed on Spark, and aggregated into customer "statistics" table. If the first workflow didn't execute successfully, the second one shouldn't be started.
2. Call ELT on Usage data if ETL on Customers data executed successfully

  1. Build an automated construction that will control whether to execute 04.2_ELT_Usage (CASE Switch Start node, flow variable ports). Use ETL_failed variable to control which port is active

  2. Call the workflow 04.2_ELT_Usage and import its final status (Call Workflow Service node)

    • Use the workflow relative path & auto-adjust the ports

  3. OPTIONAL: Use the second port of the CASE Switch Start nodes (that gets active in case of failure) to notify a responsible person via email (Send Email node). To try it out, you would need to use your own credentials.


1. Call ETL on Customers data workflow and import its final status

  1. In the Webservice and S3 Credentials Configuration component, provide the username and password to the webservice and Amazon S3, the webservice URL.

  2. Call the workflow 04.1_ETL_Customers and import its final status (Call Workflow Service node)

    • Use the workflow relative path & auto-adjust the ports

    • Connect the Credentials Configuration component to the input flow variable port


Session 4 - Orchestration

Exercise 03 Orchestration

Learning objective: Learn how to orchestrate workflows and execute workflows sequentially based on the execution status


Workflow description: Once the ETL on Customers data workflow is executed successfully, i.e., customer data are accessed and anonymized, and the database table "customers" is updated, the next process - ELT on Usage data - should be triggered. In this workflow, the usage data are accessed, transformed on Spark, and aggregated into customer "statistics" table. If the first workflow didn't execute successfully, the second one shouldn't be started.


You'll find the instructions to the exercises in the yellow annotations.

Webservice and S3 Credentials Configuration
ETL_failed = 0: execute the next workflow ETL_failed = 1: stop
CASE Switch Start
Execute 04.2_ELT_Usage and import its status
Call Workflow Service
Notify the responsible preson
Send Email
Execute 04.1_ETL_Customers and import its status
Call Workflow Service
top: ELT_failed = 0 bottom: ELT_failed = 1
CASE Switch Start
Notify the responsible preson
Send Email

Nodes

Extensions

Links