Icon

AP-22457_​ParallelChunk_​LoopPerformance

AP-22457: Parallel Chunk Loop (End) is unnecessarily slow due to synchronous data write (columnar backend)

The issue was an unnecessary synchronization when writing the output in the Parallel Chunk End when the "Columnar Backend" was set on the workflow.

Performance comparisons for 50M rows (data generator), with a par-chunker containing a row filter removing about 2/3 of the rows:

Runtime comparison (on my system):
- Parallel Chunk, 5.2.3 : 202s
- Parallel Chunk, 5.3 Nightly: 65s
- Plain Row Filter: 35s
(no par-chunker, just for reference)

https://forum.knime.com/t/any-ideas-when-parrallel-chunk-loop-end-performance-will-be-resolved/78568


AP-22457: Parallel Chunk Loop (End) is unnecessarily slow due to synchronous data write (columnar backend)

Before the fix (AP-22457) this part of the workflow would take long to run (202s, on my system). After the change, the execution completes in 65s. It takes ~2x as long as a single stream execution (below) since an extra data copy is required (to be addressed in version 5.3).

Baseline comparison.

50M rows
Data Generator
Parallel Chunk Start
Row Filter (legacy)
Parallel Chunk End
Row Filter (legacy)
Timer Start
Timer End
Timer End
Timer Start
Table Difference Checker

Nodes

Extensions

Links