Icon

03 Production data pipeline - Workflow invocation - Exercise

This workflow:
Step 3 : Write the transformed data to the production database

  1. The Connector and DB Writer node are pre-configured. Execute them.


Note. When deploying on KNIME Business Hub, you'll use the PostgreSQL database.



Step 2 : Call the Data Transformation workflow

  1. In the Call Workflow Service node:

    • Provide the workflow relative path to the Data Transformation workflow

    • ../00 Deployment workflows/Data_Transformation

    • Adjust node ports



Step 1 : Extract raw orders data from Amazon S3

  1. In the Amazon Authentication node:

    • Provide the Access Key ID and Secret Key provided by the trainers

    • Select the eu-west-1 as a region

  2. The Amazon S3 Connector and CSV Reader nodes are already configured to read the CSV file orders_oct_21-27.csv from folder l3-de in knime-education-l4-de-eu-west bucket. Execute them.


Note: In general, you can browse though different folders on S3. In this course, provided credentials have read-only permissions, therefore, browsing is limited to the knime-education-l4-de-eu-west bucket.




Call the Data Transformation workflow
Recent Week data

Session 1 - Preparing a Data Pipeline for Deployment

Exercise 03 Production data pipeline - Workflow invocation

Learning objective: Learn how to build the data pipeline for production


Workflow description: This workflow:

  • Reads the data from real data sources, e.g., weekly orders data stored in Amazon S3,

  • Calls the developed and tested Data Transformation workflow to transform the raw orders data,

  • Loads the transformed data to the production database


You'll find the instructions to the exercises in the yellow annotations.

Production Database


Connect to S3 Specify working directory
Amazon S3 Connector
Append:orders_cleaned
DB Writer
Authenticate against Amazon
Amazon Authenticator
SQLite Connector
Read orders datafrom Amazon S3
CSV Reader

Nodes

Extensions

Links