0 ×

Parquet Writer

StreamableKNIME BigData File Format Extension version 4.2.0.v202007031129 by KNIME AG, Zurich, Switzerland

This node writes the KNIME data table into a Parquet file. The Parquet file can be written locally or on a remote file system.

For remote connections, the writer stores chunks locally and uploads them in parallel into a folder with the given name. The chunks size and the number of chunks that are allowed on local disc, are configurable in the "Chunk Upload" tab which is only enabled for remote file systems.

Options

Output location: A path on the local machine or remote server where the file is written.
Overwrite: If this option is selected the file will be overwritten if it exists
Check directory content: Enabled when using a remote connection and enabling the overwrite option. With this option enabled the node will fail if the content of the directory contains files that do not fit the file format suffix
File Compression: The compression codec used to write the Parquet file.

Chunk Upload

Chunk size in MB: The maximum size for an individual chunk.
Number of local chunks: The number of chunks that can be stored temporarily on disc, while the next chunk is written.

Output Type Mapping

Change the KNIME to Parquet type mapping configuration for subsequent nodes by selecting a Parquet type to the given KNIME Type. The dialog allows you to add new or change existing type mapping rules.

Mapping by Name: Columns that match the given name (or regular expression) and KNIME type will be mapped to the specified Parquet type.
Mapping by Type: Columns that match the given KNIME type will be mapped to the specified Parquet type.

Input Ports

: An optional connection to the remote file system
: The data table that should be written

Best Friends (Incoming)

Column Filter (7 %) Streamable
Concatenate (6 %) Streamable
Table Row to Variable (6 %)
CSV Reader (6 %)
Row Filter (5 %) Streamable
Show all 105 recommendations

Best Friends (Outgoing)

Parquet Reader (21 %) Streamable
Parquet to Spark (14 %)
Variable Loop End (11 %)
Send Email (8 %)
Component Output (7 %)
Show all 28 recommendations

Workflows

06_Connecting_to_Databricks (KNIME Hub)
08_Learning_Asociation_Rule_for_Next_Restaurant_Prediction (KNIME Hub)
kn_automl_h2o_classification_python (KNIME Hub)
kn_automl_h2o_classification_python_vtreat (KNIME Hub)
kn_automl_h2o_classification_r (KNIME Hub)
Show all 21 workflows

Installation

To use this node in KNIME, install KNIME Extension for Big Data File Formats from the following update site:

KNIME 4.2

A zipped version of the software site can be downloaded here. Read our FAQs to get instructions about how to install nodes from a zipped update site.

Wait a sec! You want to explore and install nodes even faster? We highly recommend our NodePit for KNIME extension for your KNIME Analytics Platform.

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.