Icon

04.3_​Orchestration

04.3 Orchestration_solution

Once the workflow the ETL on Customers data is executed successfully, i.e., customer data are accessed and anonymized, and the database table "customers" is updated, the next process - ELT on Usage data - should be triggered. In the next workflow, the usage data are accessed, transformed on Spark, and aggregated into customer "statistics" table. If the first workflow didn't execute successfully, the second one shouldn't be started.



ExercisesAll the tasks can be found in the yellow annotations and the yellow components.Execute 04.0_Reset_DB&Big_Data_Environment before running this workflow 1 Call ETL on Customers data workflow and importits final status1. In the Credentials Configuration component, provide the username andpassword to the webservice, Amazon S3, and PostgreSQL database.2. In the Parameters Configuration component, provide the webserviceURL, and the DB parameters (hostname, database name, port).3. Call the workflow 04.1_ETL_Customers and import its final status (CallWorkflow Service node)- Use the workflow relative path & auto-adjust the ports- Connect the Credentials Configuration component to the input flowvariable port 2 Call ELT on Usage data if ETL on Customers data executedsuccessfully1. Build an automated construction that will control whether to execute 04.2_ELT_Usage (CASE SwitchStart node, flow variable ports). Use ETL_failed variable to control which port is active2. Call the workflow 04.2_ELT_Usage and import its final status (Call Workflow Service node)- Use the workflow relative path & auto-adjust the ports3. OPTIONAL: Use the second port of the CASE Switch Start nodes (that gets active in case of failure) tonotify a responsible person via email (Send Email node). To try it out, you would need to use your owncredentials. Session 4 OrchestrationExercise 04.3 Orchestration ETL_failed = 0: executethe next workflowETL_failed = 1: stopExecute04.1_ETL_Customersand import its statusExecute04.2_ELT_Usageand import its statusNotify theresponsible presonNotify theresponsiblepresontop: ELT_failed = 0bottom: ELT_failed = 1 CredentialsConfiguration CASE Switch Start Call WorkflowService Call WorkflowService Send Email Send Email CASE Switch Start ParametersConfiguration ExercisesAll the tasks can be found in the yellow annotations and the yellow components.Execute 04.0_Reset_DB&Big_Data_Environment before running this workflow 1 Call ETL on Customers data workflow and importits final status1. In the Credentials Configuration component, provide the username andpassword to the webservice, Amazon S3, and PostgreSQL database.2. In the Parameters Configuration component, provide the webserviceURL, and the DB parameters (hostname, database name, port).3. Call the workflow 04.1_ETL_Customers and import its final status (CallWorkflow Service node)- Use the workflow relative path & auto-adjust the ports- Connect the Credentials Configuration component to the input flowvariable port 2 Call ELT on Usage data if ETL on Customers data executedsuccessfully1. Build an automated construction that will control whether to execute 04.2_ELT_Usage (CASE SwitchStart node, flow variable ports). Use ETL_failed variable to control which port is active2. Call the workflow 04.2_ELT_Usage and import its final status (Call Workflow Service node)- Use the workflow relative path & auto-adjust the ports3. OPTIONAL: Use the second port of the CASE Switch Start nodes (that gets active in case of failure) tonotify a responsible person via email (Send Email node). To try it out, you would need to use your owncredentials. Session 4 OrchestrationExercise 04.3 Orchestration ETL_failed = 0: executethe next workflowETL_failed = 1: stopExecute04.1_ETL_Customersand import its statusExecute04.2_ELT_Usageand import its statusNotify theresponsible presonNotify theresponsiblepresontop: ELT_failed = 0bottom: ELT_failed = 1 CredentialsConfiguration CASE Switch Start Call WorkflowService Call WorkflowService Send Email Send Email CASE Switch Start ParametersConfiguration

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Lada.Rudnitckaia
Created at: 2021-10-14
On NodePit since: 2023-04-06
Last update: 2024-05-01
Created with KNIME version: v4.7.1
Tags: Best practicesData engineerData engineeringEducationorchestrationworkflow dependenciesworkflow service

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!