Icon

02 Production data pipeline - Ad hoc execution, versioning, and scheduling - Exercise

This workflow:
Step 1 : Extract raw orders data from a raw data source and call the Data Transformation workflow on the KNIME Business Hub

Note. In real case scenario you would access raw data from a data source such as data lake or cloud file system, a web service, an OLTP database, etc. We don't host data on such a source for this exercise. For the exercise, you will access the file from a space on KNIME Business Hub. If you had to access the same file from, e.g., Amazon S3, you would need to simply adjust the connection.


  1. Access the orders_oct_21-27.csv file residing on KNIME Business Hub (see exercise 03 in part 1)

  2. Connect to KNIME Business Hub space with the Space Connector node (see exercise 01 in part 2)

  3. Call the Data_Transformation workflow saved to KNIME Business Hub in the previous exercise with the Call Workflow Service node

    • Select the latest version and the valid execution context



Step 2 : Set up data loading and upload the workflow to KNIME Business Hub

Note. From a workflow deployed on KNIME Business Hub, you can connect to a database that is hosted on a remote server. We don't host such a database for this exercise and use a file as a placeholder for data loading.


  1. Append new orders to the development database (use orders_cleaned_dev.csv as a placeholder)

  2. Save this workflow and upload it to your user space on KNIME Business Hub*


*Make sure to reset all the nodes



Recent Week data
Write to Database

Learning objective: Learn how to execute, version, and schedule a workflow on the KNIME Business Hub


Workflow description: This workflow:

  • Reads the raw data from a remote data source, e.g., weekly orders data stored on KNIME Business Hub,

  • Calls the developed and tested Data Transformation workflow to transform the raw orders data,

  • Loads the transformed data to the production repository,

  • Besides, this workflow should be scheduled to run regularly and update the database table.


You'll find the instructions to the exercises in the yellow annotations.

Part 2 - Introduction to KNIME Business Hub

Exercise 02 Production data pipeline - Ad-hoc execution, versioning & scheduling

Call the Data Transformation workflow
Step 3 : Test the production data pipeline with ad-hoc execution

  1. Perform ad-hoc execution of this workflow on KNIME Business Hub to test that it executes successfully on the KNIME Business Hub. On the workflow page on KNIME Business Hub:

    • Press the Run button to execute the workflow immediately

    • In the appeared menu, click the Run button once more

    • See if the workflow executed successfully and visualizes the uploaded data

  2. Create a version of the production data pipeline workflow:

    • On the workflow page, click History

    • In the new menu, click Create version and name it as Hub execution test on the dev database

Step 4 : Finalize the workflow and schedule it to run every week

  1. Finalize the workflow:

    1. In this workflow, change the Development database to Production database (use orders_cleaned_prod.csv as a placeholder)

    2. Reupload and overwrite the workflow on KNIME Business Hub

    3. Create a new version on KNIME Business Hub, and name it as Production ready

  2. Schedule the latest deployed version of the workflow:

    • On the workflow page, click the Deploy button

    • Click on Create schedule button to schedule the workflow to run automatically

      • Provide the Deployment name as UserXX-schedule

      • Select the latest version Production ready

      • Select a valid execution context

      • Schedule the workflow to run at 19:00 every Monday

      • Click the Create button to create the schedule

Call Workflow Service
Read orders datafrom data source
CSV Reader
DB Writer results
Placeholder for a realraw data source
Space Connector
Space Connector
Databaseplaceholder
Data loading

Nodes

Extensions

Links