0 ×

Spark to Avro

KNIME Extension for Apache Spark core infrastructure version 4.3.1.v202101261633 by KNIME AG, Zurich, Switzerland

This node supports the path flow variable. For further information about file handling in general see the File Handling Guide.

Writes a Spark data to Avro.

Notice: This feature requires at least Apache Spark 1.5.

Options

Settings

Write to

Shows the connected file system.

Folder

Enter the output path. The required syntax of a path depends on the connected file system. The node description of the respective connector node describes the required path format. You can also choose a previously selected file from the drop-down list, or select a location from the "Browse..." dialog. Note that browsing is disabled in some cases:

Browsing is disabled if the connector node hasn't been executed since the workflow has been opened. (Re)execute the connector node to enable browsing.

The location can be exposed as or automatically set via a path flow variable.

Create missing folders

Select if the folders of the selected output location should be created if they do not already exist. If this option is unchecked, the node will fail if a folder does not exist.

If exists

Specify the behavior of the node in case the output file already exists.

Overwrite: Will replace any existing file.
Append: Append to existing files or create new one if output path does not exists.
Ignore: Does nothing and keeps the output path unchanged.
Fail: Will issue an error during the node's execution (to prevent unintentional overwrite).

Driver

Upload data source driver or depend on cluster side provided driver.

Partitions

Columns: Select the columns to partition on.
Overwrite partitions count: This can be useful to reduce output file count to e.g. one file.
Warning: This might result in serious performance issues on huge data sets. Use with caution!
See Spark documentation for more informations.

Input Ports

: Spark compatible connection (HDFS, WebHDFS, HttpFS, S3, Blob Storage, ...)
: Spark DataFrame/RDD

Installation

To use this node in KNIME, install KNIME Extension for Apache Spark from the following update site:

KNIME 4.3

A zipped version of the software site can be downloaded here.

You don't know what to do with this link? Read our NodePit Product and Node Installation Guide that explains you in detail how to install nodes to your KNIME Analytics Platform.

Wait a sec! You want to explore and install nodes even faster? We highly recommend our NodePit for KNIME extension for your KNIME Analytics Platform. Browse NodePit from within KNIME, install nodes with just one click and share your workflows with NodePit Space.

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.