Icon

03b OPTIONAL Production data pipeline - Trigger - Exercise

This workflow:
Step 2 : Call the Data Transformation workflow on the KNIME Hub

  1. Connect to KNIME Business Hub space with the Space Connector node (see exercises in session 2)

  2. Call the Inspection_Results_Transformation_Workflow workflow saved to KNIME Business Hub in the previous exercise with the Call Workflow Service node

    • Select the latest version and the valid execution context



Step 1 : Extract raw inspection data from KNIME Business Hub and GitHub when a new inspection report appears in a KNIME Business Hub space

  1. Use the Container Input (Repository Event) node to import the event details when the workflow gets triggered (when a new report gets uploaded)*

  2. Connect to KNIME Business Hub with the Space Connector node:

    • Set knime-edu-hub as the Hub mountpoint and select your exercise space

    • Set the working directory as /Inspection

    • Connect the Container Input (Repository Event) and the Space Connector nodes to the PDF report text metanode


*During the development of this workflow, we use example event details.



Learning objective: Learn how to trigger workflow execution on the KNIME Business Hub


Workflow description: This workflow:

  • Reads the data from real data sources, e.g., new inspection reports once they appear in KNIME Business Hub space,

  • Calls the developed and tested Data Transformation workflow to blend and transform raw inspection files and inspection reviews,

  • Loads the transformed data to the production database,

  • Besides, this workflow should be triggered each time when a new report appears in the selected KNIME Business Hub space.


You'll find the instructions to the exercises in the yellow annotations.

Extract
Transform
Load
Step 4 : Deploy the workflow as a trigger

  1. On the workflow page on KNIME Business Hub, create a version of your workflow

  2. Create the trigger deployment:

    • Click Deploy and Trigger buttons

    • Provide the deployment name as UserXX-triggered-execution, select the latest created version and a valid execution context

    • Configure the trigger options:

      • Space: your private ExerciseSpace-UserXX

      • Item type: File

      • File type: Data file

      • Action: Added

      • Additional filtering: By subfolder

      • Subfolder: /Inspection

      • Click the Create button

  3. Add one of the PDF files to your Inspection folder in your space

    • You can find the PDF files in the exercises data folder ../../../../data/Inspection

    • Right click and select Upload to Hub

    • The execution of this workflow should get triggered by this event

Session 2 - Introduction to KNIME Business Hub

Exercise 03b (Optional) Production data pipeline - Triggered execution

Step 3 : Set up the database connection and upload the workflow to KNIME Business Hub

  1. Connect to the production database and append new orders data:

    • In the PostgreSQL Connection component:

      • Provide the credentials, hostname, database and schema names as provided by the course trainers

      • Provide the table name as inspection_cleaned

    • The DB Writer node is pre-configured and is controlled by the variables coming from the component.

  2. Save this workflow and upload it to your user space on KNIME Business Hub*


*Make sure to reset all the nodes



Production Database


Preparation Step : Upload an example PDF file to your space on Hub to be able to execute this workflow

Before configuring this workflow:

  1. Connect to your user space on KNIME Business Hub and run the Transfer Files node to create an Inspection folder in your user space and upload an example PDF report to it.

  2. Once you executed these two nodes, delete them!



Append:inspection_cleaned
DB Writer
Space Connector
Inspection folder inyour user space
Space Connector
PDF report text
GitHub repo
Downlaod from GitHub Repo
Call Workflow Service
Read Inspection Reviews
JSON Reader
Create Inspectionfolder & uploadAdriano.pdf to it
Transfer Files
Your user space
Space Connector
PostgreSQL Connection

Nodes

Extensions

Links