Input data validation

Step 1 : Empty table check

1. Execute the Extract Table Dimension node - it exports the flow variable Number Rows

2. Add the Breakpoint node and enable the breakpoint when variable Number Rows = 0

Learning objective: Learn how to create a workflow for input data validation

Workflow description: This workflow implements the input data validation and checks:

• Whether input data table is empty,

• Whether input data has all the required columns,

• Statistical properties of the input data,

• And raises error if any of the tests fails.

You'll find the instructions to the exercises in the yellow annotations.

Session 4 - Best Practices when Productionizing Data Pipelines

Exercise 01 Data validation & Statistics Process Control

Step 2 : Table structure validation

1. Add the Table Validator node between the Workflow Service Input and the Breakpoint nodes

2. In the configuration of the Table Validator node, add a new column group by double clicking on the column and drag all the other columns to this group

3. Add the Breakpoint node and enable the breakpoint for inactive branch

Step 3 : Statistical Process Control

1. Use the Math Formula node after the String to Number node to calculate the maximal price

   • Use the expression: COL_MAX($Price$)

   • Append a new column Max_price

2. Use the Rule Engine node to create an error trigger if the maximal price unexpectedly high

   • Use the expression: $Max_price$ >1000 => 1

   • Append a new column price_flag

3. Convert the price_flag to a flow variable with Table Row to Variable node

4. Add the Breakpoint node and enable the breakpoint when variable price_flag = 1

Step 4 : Upload to KNIME Business Hub

Reset the workflow and upload to your user space on KNIME Business Hub