Icon

03 Production data pipeline - Workflow invocation - Exercise

This workflow:
Step 3 : Write the transformed data to the production database

  1. Connect to the production database and append new orders data:

    • In the PostgreSQL Connection component:

      • Provide the credentials, hostname, database and schema names as provided by the course trainers

      • Provide the table name as orders_cleaned

    • The DB Writer node is pre-configured and is controlled by the variables coming from the component.



Step 2 : Call the Data Transformation workflow

  1. In the Call Workflow Service node:

    • Provide the workflow relative path to the Data Transformation workflow

    • ../00 Deployment workflows/Data_Transformation

    • Adjust node ports



Step 1 : Extract raw orders data from Amazon S3

  1. In the Amazon Authentication node:

    • Provide the Access Key ID and Secret Key provided by the trainers

    • Select the eu-west-1 as a region

  2. The Amazon S3 Connector and CSV Reader nodes are already configured to read the CSV file orders_oct_21-27.csv from folder l3-de in knime-education-l4-de-eu-west bucket. Execute them.


Note: In general, you can browse though different folders on S3. In this course, provided credentials have read-only permissions, therefore, browsing is limited to the knime-education-l4-de-eu-west bucket.




Call the Data Transformation workflow
Recent Week data

Session 1 - Preparing a Data Pipeline for Deployment

Exercise 03 Production data pipeline - Workflow invocation

Learning objective: Learn how to build the data pipeline for production


Workflow description: This workflow:

  • Reads the data from real data sources, e.g., weekly orders data stored in Amazon S3,

  • Calls the developed and tested Data Transformation workflow to transform the raw orders data,

  • Loads the transformed data to the production database


You'll find the instructions to the exercises in the yellow annotations.

Production Database


Connect to S3 Specify working directory
Amazon S3 Connector
Append:orders_cleaned
DB Writer
Authenticate against Amazon
Amazon Authenticator
Read orders datafrom Amazon S3
CSV Reader
PostgreSQL Connection

Nodes

Extensions

Links