Icon

CSV - Import Large File

<p>Import very large CSV files with CSV Reader and PyArrow</p>

URL: Import very large CSV files with CSV Reader and PyArrow - KNIME Forum (91051) https://forum.knime.com/t/how-does-knime-handle-csv-files-of-2gb/91051/5?u=mlauber71
URL: KNIME Snippets (1): Collect and Restore — or how to handle many large files and resume loops https://medium.com/low-code-for-advanced-data-science/knime-snippets-1-collect-and-restore-or-how-to-handle-many-large-files-and-resume-loops-c57795b65d7e

Import very large CSV files with CSV Reader and PyArrow

https://forum.knime.com/t/how-does-knime-handle-csv-files-of-2gb/91051/5?u=mlauber71

https://hub.knime.com/-/spaces/-/~4-Nz2crmY1OvrH_M/current-state/
Create Dummy Data
3 times
Counting Loop Start
sample.csvpipe as separator
CSV Writer
Variable Loop End
Recursive Loop Start
Table Row to Variable
import with PyArrow
Python Script
Path to URI
URL to File Path
sample.csvallow large columns
CSV Reader
*.csv
List Files/Folders
max 12 times
Recursive Loop End
Concatenate

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: m_lauber
Created at: 2025-12-23
On NodePit since: 2025-12-23
Last update: 2025-12-23
Created with KNIME version: v5.9.0
Tags: csvimportlargePyArrow

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!