Icon

Parquet - Split Data into several Files

<p>Export Data to Parquet Files split in several parts into a folder</p>

URL: forum entry https://forum.knime.com/t/parquet-file-reader-taking-a-long-time/20020/17?u=mlauber71
URL: (65232) forum entry - large files and R https://forum.knime.com/t/knime-assign-failed-request-status-data-overflow-incoming-data-too-big/65232/2?u=mlauber71
URL: MEDIUM: KNIME and R — installation across operating systems — some remarks https://medium.com/p/6494a2a498cc

Export Data to Parquet Files split in several parts into a folder

https://forum.knime.com/t/append-method-in-the-write-table-node/90022/3?u=mlauber71

https://hub.knime.com/-/spaces/-/~4-Nz2crmY1OvrH_M/current-state/
Create Dummy Data
Concatenate
Sorter
import all parquet files from the subfolder/data/test_folder_parquet/....
Parquet Reader
Counter Generation
Concatenate
import all parquet files from the subfolder/data/test_folder_parquet/....
Parquet Reader
Column Filter
/data/test_folder_parquet/...
Delete Files/Folders
the last counter plus 1
Math Formula
GroupBy
Counter Generation
Table Row to Variable
/data/test_folder_parquet/...split large data into parts=> note: in a real world scenario you shouldset the split values to a larger default1024 MB and 128 MB
Parquet Writer
test_folder_parquet/part_append.parquet
Parquet Writer
Try (Variable Ports)
Catch Errors (Var Ports)

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: m_lauber
Created at: 2019-12-17
On NodePit since: 2025-10-07
Last update: 2025-10-07
Created with KNIME version: v5.7.0
Tags: parquetimportknimereadgzip

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!