Icon

02 Data Quality Assessment

02 Data Quality Assessment
retraining-flag == 0 retraining-flag == 1 Read data from database Access the boundaries to provide to the component This workflow carries out the data quality assessment on a selected column for the chosen date interval. It reads the boundaries saved in a log and provides them to the Data Quality Assessment component.The caller workflow specifies the column to test, the end date of the considered period and a period in months (substracted from the date to get the starting date).If the model needs to be updated (retraining-flag == 1), this workflow logs the updatd boudaries, calculated according to the new training data.After each execution, the results of the data quality assessment are exported to a second log file. Calculate period to consider for the Data QualityAssessment using month-interval variable Input: Data for quality assmentOutput: retraining flag Input: Training data of theretrained modelOutput: new boundaries Append resultto log fileIn the configuration windowset boundaries and selectcolumn to be checkedboundaries_log.csvAppend new boundariesResult of the dataquality assessmentFilter lastentryuse months-periodvariableSelect columnFrom the beginningFilter bymonths intervalFiltertested-columntested-columnconsidered-period-endtested-columnmonths-interval DB Table Selector CSV Writer SQLite Connector Data QualityAssessment CSV Reader CSV Writer Format log Rule-basedRow Filter Table Rowto Variable Calculateconsidered period New BoundariesCalculation Formatboundaries log Data Access Data Access CASE Switch Start CASE Switch End Row Filter Table Rowto Variable WorkflowService Input WorkflowService Output retraining-flag == 0 retraining-flag == 1 Read data from database Access the boundaries to provide to the component This workflow carries out the data quality assessment on a selected column for the chosen date interval. It reads the boundaries saved in a log and provides them to the Data Quality Assessment component.The caller workflow specifies the column to test, the end date of the considered period and a period in months (substracted from the date to get the starting date).If the model needs to be updated (retraining-flag == 1), this workflow logs the updatd boudaries, calculated according to the new training data.After each execution, the results of the data quality assessment are exported to a second log file. Calculate period to consider for the Data QualityAssessment using month-interval variable Input: Data for quality assmentOutput: retraining flag Input: Training data of theretrained modelOutput: new boundaries Append resultto log fileIn the configuration windowset boundaries and selectcolumn to be checkedboundaries_log.csvAppend new boundariesResult of the dataquality assessmentFilter lastentryuse months-periodvariableSelect columnFrom the beginningFilter bymonths intervalFiltertested-columntested-columnconsidered-period-endtested-columnmonths-interval DB Table Selector CSV Writer SQLite Connector Data QualityAssessment CSV Reader CSV Writer Format log Rule-basedRow Filter Table Rowto Variable Calculateconsidered period New BoundariesCalculation Formatboundaries log Data Access Data Access CASE Switch Start CASE Switch End Row Filter Table Rowto Variable WorkflowService Input WorkflowService Output

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: maarit.widmann
Created at: 2021-10-14
On NodePit since: 2022-04-22
Last update: 2024-04-24
Created with KNIME version: v4.5.2
Tags: data driftsynthetic datamodel monitoring

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!