Icon

01 Development data pipeline - Integrated deployment - Exercise

This workflow reads a sample of restaurant orders data, cleans and transforms it, and loads the transformed data to the development database. Besides, it captures the data transformation segment and saves it as a separate Data Transformation workflow.
Step 2 : Write the transformed data to database

  1. Connect to development database and write the data

    • In the PostgreSQL Connection component:

      • Provide the credentials, hostname, database and schema names as provided by the course trainers

      • Provide the table name as orders_cleaned

    • The DB Writer node is pre-configured and is controlled by the variables coming from the component.



Step 1 : Capture the workflow segment using Integrated Deployment

  1. Capture the Data Transformation segment of the workflow

    • After reading the data, add the Capture Workflow Start node to begin the workflow capturing process

    • Use the Capture Workflow End node after the Summarize TimeStamp-Hour component to end the capturing process



Development Database


Data Transformation

Session 1 - Preparing a Data Pipeline for Deployment

Exercise 01 Development data pipeline - Integrated deployment

Learning objective: In this exercise you'll practice using the Integrated Deployment extension to capture the data transformation workflow segment and save it as a new workflow


Workflow description: This workflow reads a sample of restaurant orders data, cleans and transforms it, and loads the transformed data to the development database. Besides, it captures the data transformation segment and saves it as a separate Data Transformation workflow.


You'll find the instructions to the exercises in the yellow annotations.

Read Data from Recent Week

Step 3 : Save the Data Transformation workflow


  1. In the Workflow Writer node:

    • Use the workflow relative path to write the workflow to /00 Deployment workflows folder

    • Enable overwriting of the workflow

    • Use custom workflow name: Data_Transformation

    • In the Inputs and outputs tab:

      • Select Workflow Service Input and Workflow Service Output nodes for the input and output nodes to be added automatically to the newly created workflow

  2. Explore the automatically created workflow



Summarize Timestamp-Hour
"Price" Column to Number
String to Number
"Table ID" Column to Number
String to Number
Append:orders_cleaned
DB Writer
by time, restaurant
Sorter
Date&Time
String to Date&Time
Remove Duplicates
Duplicate Row Filter
Filter relevant cols
Column Filter
Sample data
CSV Reader
PostgreSQL Connection

Nodes

Extensions

Links