Icon

02 Production data pipeline - Ad hoc execution, versioning, and scheduling

This workflow:
Step 2 : Set up the database connection and upload the workflow to KNIME Business Hub

  1. Connect to the development database and append new orders data:

    • In the PostgreSQL Connection component:

      • Provide the credentials, hostname, database and schema names as provided by the course trainers

      • Provide the table name as orders_cleaned

    • The DB Writer node is pre-configured and is controlled by the variables coming from the component.

  2. Save this workflow and upload it to your user space on KNIME Business Hub*


*Make sure to reset all the nodes



Step 1 : Extract raw orders data from Amazon S3 and call the Data Transformation workflow on the KNIME Business Hub

  1. Connect to Amazon S3 (see exercise 03 in session 1)

  2. Connect to KNIME Business Hub space with the Space Connector node (see exercise 01 in session 2)

  3. Call the Data_Transformation workflow saved to KNIME Business Hub in the previous exercise with the Call Workflow Service node

    • Select the latest version and the valid execution context



Recent Week data
Write to Database

Learning objective: Learn how to execute, version, and schedule a workflow on the KNIME Business Hub


Workflow description: This workflow:

  • Reads the data from real data sources, e.g., weekly orders data stored in Amazon S3,

  • Calls the developed and tested Data Transformation workflow to transform the raw orders data,

  • Loads the transformed data to the production database,

  • Besides, this workflow should be scheduled to run regularly and update the database table.


You'll find the instructions to the exercises in the yellow annotations.

Session 2 - Introduction to KNIME Business Hub

Exercise 02 Production data pipeline - Ad-hoc execution, versioning & scheduling

Production Database


Call the Data Transformation workflow
Step 3 : Test the production data pipeline with ad-hoc execution

  1. Perform ad-hoc execution of this workflow on KNIME Business Hub to test that it executes successfully on the KNIME Business Hub. On the workflow page on KNIME Business Hub:

    • Press the Run button to execute the workflow immediately

    • In the appeared menu, click the Run button once more

    • See if the workflow executed successfully and visualizes the uploaded data

  2. Create a version of the production data pipeline workflow:

    • On the workflow page, click History

    • In the new menu, click Create version and name it as Hub execution test on the dev database

Step 4 : Finalize the workflow and schedule it to run every week

  1. Finalize the workflow:

    1. In this workflow, change the Development database to Production database

    2. Reupload and overwrite the workflow on KNIME Business Hub

    3. Create a new version on KNIME Business Hub, and name it as Production ready

  2. Schedule the latest deployed version of the workflow:

    • On the workflow page, click the Deploy button

    • Click on Create schedule button to schedule the workflow to run automatically

      • Provide the Deployment name as UserXX-schedule

      • Select the latest version Production ready

      • Select a valid execution context

      • Schedule the workflow to run at 19:00 every Monday

      • Click the Create button to create the schedule

Call Workflow Service
Connect to S3 Specify working directory
Amazon S3 Connector
Append:orders_cleaned
DB Writer
Space Connector
DB Writer results
Read orders datafrom Amazon S3
CSV Reader
Authenticate against Amazon
Amazon Authenticator
Development DBfor tests on KBHProduction DBafter the tests
PostgreSQL Connection

Nodes

Extensions

Links