Session 4 - Introduction to Data Engineering

Exercise 02 Loading Tables to Database and Logging Events

Learning objective: In this exercise you will practice the concept of in-database processing and creation of logs that can register all the events.

Workflow description: This workflow reads data from two separate tables in databases, performs in-database processing on these tables to create a report on the store level. The logging component at the end logs the important fields like execution status, time etc

• In the first segment of the workflow, the DB tables - Orders and License Renewal are read and then in-database processing is performed to create a store-level report

• The Try and Catch block makes sure the workflow execution never fails and the Catch Error node provides information on whether any nodes in the block have failed during the execution

• At the end, all the information from the Catch error node is used to log the events using the Logging Component

You will find the instructions to the exercises in the yellow annotations.

Activity III. Logging the Events

1. Set up the success and failure status messages using a flow variable with the Variable Creator node.

   • Open the Configuration dialog of the Variable Creator node on the top branch and create a new flow variable with
     Type: String
     Variable Name: Status
     Value: Success
     Connect the flow variable output to the top input of the Catch Errors (Var Ports) node

   • Open the Configuration dialog of the Variable Creator node on the bottom branch and create a new flow variable with
     Type: String
     Variable Name: Status
     Value: Failure
     Connect the flow variable output to the bottom input of the Catch Errors (Var Ports) node

2. Set up the Logging component to log events. Further instructions are provided in the Logging component

Detect Success And Failure Status

Logging the Metadata

Export the Results