Icon

03 Data pipeline orchestration

<p>Once the ETL on new customer data workflow is executed successfully, i.e., customer data are extracted and anonymized, and the database table "customers" is updated, the next process - ELT on website usage data - should be triggered. If the first workflow didn't execute successfully, the second one shouldn't be started.</p>
Step 4. Call ELT on website usage data if ETL on new customer data executed successfully

  1. Build an automated construction that will control whether to execute 02.2 ELT on website usage data - transform big data on cloud and Spark workflow (CASE Switch Start node, flow variable ports). Use ETL_failed variable to control which port is active

  2. Call the workflow 02.2 ELT on website usage data - transform big data on cloud and Spark and import its final status (Call Workflow Service node)

    • Use the workflow relative path & auto-adjust the ports]


Step 3. Call ETL on new customer data workflow and import its final status

  1. In the Retrieve credentials component, provide the username and password to access the Amazon S3 bucket and the web service (similar to exercise workflow 01)

  2. Call the workflow 01 ETL on new customer data - extract, transform, and load S3 and web service data and import its final status (Call Workflow Service node)

    • Use the workflow relative path & auto-adjust the ports

    • Connect the Retrieve credentials component to the input flow variable port


Exercise workflow 03

Data pipeline orchestration

Learning objective: Orchestrate and execute workflows conditionally based on the execution status.


Workflow description: Once the ETL on new customer data workflow is executed successfully, i.e., customer data are extracted and anonymized, and the database table "customers" is updated, the next process - ELT on website usage data - should be triggered. If the first workflow didn't execute successfully, the second one shouldn't be started.


You'll find the instructions to the exercises in the yellow annotations.

Step 1. Finalize workflows 01 and 02.2

  1. Open the workflow 01 ETL on new customer data - extract, transform, and load S3 and web service data and follow the instructions in steps 5-8

  2. Open the workflow 02.2 ELT on website usage data - transform big data on cloud and Spark and follow the instructions in steps 3-4

Step 2. Reset the data storages

In the previous exercises, you loaded data to both the database and the big data platform and closed the connections. In this exercise, you'll load the same data but in an automated way via orchestration. Therefore, reset the data storages to its initial state to avoid duplicates and to recreate connections.

  • Disconnect the Active Branch Inverter node and execute the Reset component.

  • Connect the Active Branch Inverter node to the Reset component again to keep it disabled in this workflow.


OPTIONAL: Use the second port of the CASE Switch Start nodes (that gets active in case of failure) to notify a responsible person via email (Send Email node). To try it out, you would need to use your own credentials.


Reset
Execute02.2 ELT on website usage data - transform big data on cloud and Sparkand import its status
Call Workflow Service
Active Branch Inverter
Retrieve credentials
ETL_failed = 0: execute the next workflow ETL_failed = 1: stop
CASE Switch Start
Notify theresponsible person
Send Email
top: ELT_failed = 0 bottom: ELT_failed = 1
CASE Switch Start
Notify theresponsible person
Send Email
Execute01 ETL on new customer data - extract, transform, and load S3 and web service dataand import its status
Call Workflow Service

Nodes

Extensions

Links