Icon

04.3_​Orchestration

04.3 Orchestration_exercise

Once the workflow the ETL on Customers data is executed successfully, i.e., customer data are accessed and anonymized, and the database table "customers" is updated, the next process - ELT on Usage data - should be triggered. In the next workflow, the usage data are accessed, transformed on Spark, and aggregated into customer "statistics" table. If the first workflow didn't execute successfully, the second one shouldn't be started.



ExercisesAll the tasks can be found in the yellow annotations and the yellow components.Execute 04.0_Reset_DB&Big_Data_Environment before running this workflow Session 4 OrchestrationExercise 04.3 Orchestration 1 Call ETL on Customers data workflow and importits final status1. In the Credentials Configuration component, provide the username andpassword to the webservice, Amazon S3, and PostgreSQL database.2. In the Parameters Configuration component, provide the webserviceURL, and the DB parameters (hostname, database name, port).3. Call the workflow 04.1_ETL_Customers and import its final status (CallWorkflow Service node)- Use the workflow relative path & auto-adjust the ports- Connect the Credentials Configuration component to the input flowvariable port 2 Call ELT on Usage data if ETL on Customers data executedsuccessfully1. Build an automated construction that will control whether to execute 04.2_ELT_Usage (CASE SwitchStart node, flow variable ports). Use ETL_failed variable to control which port is active2. Call the workflow 04.2_ELT_Usage and import its final status (Call Workflow Service node)- Use the workflow relative path & auto-adjust the ports3. OPTIONAL: Use the second port of the CASE Switch Start nodes (that gets active in case of failure) tonotify a responsible person via email (Send Email node). To try it out, you would need to use your owncredentials. CredentialsConfiguration ParametersConfiguration ExercisesAll the tasks can be found in the yellow annotations and the yellow components.Execute 04.0_Reset_DB&Big_Data_Environment before running this workflow Session 4 OrchestrationExercise 04.3 Orchestration 1 Call ETL on Customers data workflow and importits final status1. In the Credentials Configuration component, provide the username andpassword to the webservice, Amazon S3, and PostgreSQL database.2. In the Parameters Configuration component, provide the webserviceURL, and the DB parameters (hostname, database name, port).3. Call the workflow 04.1_ETL_Customers and import its final status (CallWorkflow Service node)- Use the workflow relative path & auto-adjust the ports- Connect the Credentials Configuration component to the input flowvariable port 2 Call ELT on Usage data if ETL on Customers data executedsuccessfully1. Build an automated construction that will control whether to execute 04.2_ELT_Usage (CASE SwitchStart node, flow variable ports). Use ETL_failed variable to control which port is active2. Call the workflow 04.2_ELT_Usage and import its final status (Call Workflow Service node)- Use the workflow relative path & auto-adjust the ports3. OPTIONAL: Use the second port of the CASE Switch Start nodes (that gets active in case of failure) tonotify a responsible person via email (Send Email node). To try it out, you would need to use your owncredentials. CredentialsConfiguration ParametersConfiguration

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Lada.Rudnitckaia
Created at: 2021-10-14
On NodePit since: 2023-04-06
Last update: 2024-05-02
Created with KNIME version: v4.7.1
Tags: Best practicesData engineerData engineeringEducationorchestrationworkflow dependenciesworkflow service

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!