Icon

Write Table Challenge vs. CSV Writer Conveniency

CSVPro- Human readable- Ingestible by other programms- Append to existing file possible- Whilst named CSV, it can read and write all text based file typesCon- Requires data transformation (i.e. Collection, HTML, XML to String) as CSVonly accepts String, Int or Doubles- Possibility of file or data corruption due to Quote handling (i.e. for HTML)- Column Type is lost (List or Set becomes String)- Possibility of Column Type incompatibility - Slow Read due to reinterpration of column types TablePro- Optimized for Knime- Performant- Column Types preserved- And certainly moreCon- Appending data requires an unnecessarily complex, compared to CSV Writer,soltuion- Reading, despite Column Types are preserved, is still slow compared to CSVReader- Not human readable, as versitile or intercompatible as CSV Nodes- Data and Variable Pollution due to conversion from Path to String (see orangebox below)- Complex and therefore failure prone and requiring significnatly moremaintenance Solution #1: "Divide et impera"1. List all files of type table2. Split results by keeping desired file3. Split based on filter result3.0 File exists: Append + Overwrite3.1 File does not exist: Write Solution #2: Leverage default node behavior1. Try to write, fail if exists2. Invert Active Branch to trigger append-process2.1 Read table file2.2 Concat new with existing data2.3 Overwrite table file Issue / InconvenienceyFilter does not supportPath Type SuggestionEnable to append data to existing table file Personal noteThere is a (not yet available) solution to use CSVNodes and ensure to preserve column types.Status is WIP. Corresponding forum thread is: https://forum.knime.com/t/column-expressions-guidance-dynamically-set-column-type/47003 Corresponding Forum Posthttps://forum.knime.com/t/feature-request-table-writer-append-display-of-current-complexity-x/58556 Initial dataSimulateadditional datato be appended tooriginal dataJustpass throughFails astable already existsAppend Option unavailable!List allfiles of typetableUsing Split tosimulate handling ofmultiple filesTOPTable file exists--> Append ProcessBOTTOMWrite fileInverse concatto add most recentat the endOverwriteexisting filewith new dataInitailly createtable file1st IterationRead failstrigering catchbranchActivate fallbackif table fileswas not created yet(1st iteration)Appendnew dataEnsure via varaibleconnection to write file(Order of Execution)EnsureOrder of ExecutionOverwritetable fileInitailly createtable fileTest Data Generator CSV Writer Table Writer Test Data Generator CSV Writer Merge Variables Table Writer Create Temp Folder List Files/Folders Convert complexdata to String Convert complexdata to String Row Splitter Path to String(Variable) Path to String Empty Table Switch Table Reader Table Rowto Variable Concatenate Table Writer Table Writer Table Reader Catch Errors(Data Ports) Active BranchInverter Concatenate Cache Cache Try (Data Ports) Table Writer Table Writer CSVPro- Human readable- Ingestible by other programms- Append to existing file possible- Whilst named CSV, it can read and write all text based file typesCon- Requires data transformation (i.e. Collection, HTML, XML to String) as CSVonly accepts String, Int or Doubles- Possibility of file or data corruption due to Quote handling (i.e. for HTML)- Column Type is lost (List or Set becomes String)- Possibility of Column Type incompatibility - Slow Read due to reinterpration of column types TablePro- Optimized for Knime- Performant- Column Types preserved- And certainly moreCon- Appending data requires an unnecessarily complex, compared to CSV Writer,soltuion- Reading, despite Column Types are preserved, is still slow compared to CSVReader- Not human readable, as versitile or intercompatible as CSV Nodes- Data and Variable Pollution due to conversion from Path to String (see orangebox below)- Complex and therefore failure prone and requiring significnatly moremaintenance Solution #1: "Divide et impera"1. List all files of type table2. Split results by keeping desired file3. Split based on filter result3.0 File exists: Append + Overwrite3.1 File does not exist: Write Solution #2: Leverage default node behavior1. Try to write, fail if exists2. Invert Active Branch to trigger append-process2.1 Read table file2.2 Concat new with existing data2.3 Overwrite table file Issue / InconvenienceyFilter does not supportPath Type SuggestionEnable to append data to existing table file Personal noteThere is a (not yet available) solution to use CSVNodes and ensure to preserve column types.Status is WIP. Corresponding forum thread is: https://forum.knime.com/t/column-expressions-guidance-dynamically-set-column-type/47003 Corresponding Forum Posthttps://forum.knime.com/t/feature-request-table-writer-append-display-of-current-complexity-x/58556 Initial dataSimulateadditional datato be appended tooriginal dataJustpass throughFails astable already existsAppend Option unavailable!List allfiles of typetableUsing Split tosimulate handling ofmultiple filesTOPTable file exists--> Append ProcessBOTTOMWrite fileInverse concatto add most recentat the endOverwriteexisting filewith new dataInitailly createtable file1st IterationRead failstrigering catchbranchActivate fallbackif table fileswas not created yet(1st iteration)Appendnew dataEnsure via varaibleconnection to write file(Order of Execution)EnsureOrder of ExecutionOverwritetable fileInitailly createtable fileTest Data Generator CSV Writer Table Writer Test Data Generator CSV Writer Merge Variables Table Writer Create Temp Folder List Files/Folders Convert complexdata to String Convert complexdata to String Row Splitter Path to String(Variable) Path to String Empty Table Switch Table Reader Table Rowto Variable Concatenate Table Writer Table Writer Table Reader Catch Errors(Data Ports) Active BranchInverter Concatenate Cache Cache Try (Data Ports) Table Writer Table Writer

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: mw
Created at: 2023-01-06
On NodePit since: 2023-01-07
Last update: 2024-04-27
Created with KNIME version: v4.7.0

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!