Icon

wkflw_​dd_​check_​lz

1. Source file checksFile-name parsing:<market>-<ss prefix>-<file type index>-<ingestion type code>-<storage area>-<layout type code><market>-<ss prefix>-<file type index>-<ingestion type code>-<storage area>-<layout type code>-cntSample data filesRO-MSD-2-DBBTCH-LZ-1-20220127174355.txtRO-MSD-2-DBBTCH-LZ-1-20220127174355-cnt.txtRO,MSD => Ref-table-1 (market, ss-prefix)2 => Ref-table-5 (File-Type-code)DBBTCH => Ref-table-2 (ingestion-Type-code)LZ => Ref-table-1(Storage-area)1 => Ref-table-3 (Layout-Type-code)Acceptance criteria: All the parts of the file name delimited by "-" should match with the main column of thereference tables mentioned against each part.2. Data Dictionary checksif (Ref-table-3 (Layout-Type-code)) == 1{ CSV file:first line = header => read the field names from row-1Acceptance Criteria for field-name: Field-names from the header should match field names from the Ref-table-5get the length of each fieldAcceptance Criteria for field-length: field length computed should not exceed field length in the Ref-table-5 Ref_table_3Ref-table-3 (Layout-Type-code)Node 31Node 32Node 33Node 34Node 42Node 43 Table Reader ReferenceRow Filter String Manipulation DuplicateRow Filter Column Filter Rule-basedRow Splitter meta_conn_s3_bkt_lz String Widget meta_dwnld_extract_split_filename_lz 1. Source file checksFile-name parsing:<market>-<ss prefix>-<file type index>-<ingestion type code>-<storage area>-<layout type code><market>-<ss prefix>-<file type index>-<ingestion type code>-<storage area>-<layout type code>-cntSample data filesRO-MSD-2-DBBTCH-LZ-1-20220127174355.txtRO-MSD-2-DBBTCH-LZ-1-20220127174355-cnt.txtRO,MSD => Ref-table-1 (market, ss-prefix)2 => Ref-table-5 (File-Type-code)DBBTCH => Ref-table-2 (ingestion-Type-code)LZ => Ref-table-1(Storage-area)1 => Ref-table-3 (Layout-Type-code)Acceptance criteria: All the parts of the file name delimited by "-" should match with the main column of thereference tables mentioned against each part.2. Data Dictionary checksif (Ref-table-3 (Layout-Type-code)) == 1{ CSV file:first line = header => read the field names from row-1Acceptance Criteria for field-name: Field-names from the header should match field names from the Ref-table-5get the length of each fieldAcceptance Criteria for field-length: field length computed should not exceed field length in the Ref-table-5 Ref_table_3Ref-table-3 (Layout-Type-code)Node 31Node 32Node 33Node 34Node 42Node 43 Table Reader ReferenceRow Filter String Manipulation DuplicateRow Filter Column Filter Rule-basedRow Splitter meta_conn_s3_bkt_lz String Widget meta_dwnld_extract_split_filename_lz

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: ISRAR
Created at: 2022-02-03
On NodePit since: 2022-02-18
Last update: 2024-11-18
Created with KNIME version: v4.5.0

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!