PySpark Script (1 to 2)

This node allows you to execute Python code on Spark (See PySpark documentation). The code has to put the desired output in the data frame with the name resultDataFrame1 and resultDataFrame2

Options

PySpark Script Node

Flow Variable List
The list contains the flow variables that are currently available at the node input. Double clicking any of the entries will insert the respective identifier at the current cursor position (replacing the selection, if any).
Snippet text area

Enter your Python code here.

The SparkSession can be accessed via the global variable spark. The input Dataset<Row> can be accessed via the method input parameter dataFrame1. The output Dataset<Row> must be called resultDataFrame1 and resultDataFrame2.

The editor optionally provides autocompletion (CTRL + Space) using the local Python installation, if the KNIME Python Extension and the Python module jedi is installed. The path to the Python executable has to be configured in Preferences → KNIME → Python.
Note: The completion suggestions are made based on the local Python environment, which may differ from the Python setup in the cluster!

Input Ports

Icon
First input Spark DataFrame.

Output Ports

Icon
Result Spark DataFrame.
Icon
Result Spark DataFrame.

Popular Predecessors

Popular Successors

Views

This node has no views

Workflows

Links

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.

Installation

To use this node in KNIME, install the extension KNIME Extension for Apache Spark (legacy) from the below update site following our NodePit Product and Node Installation Guide:

v5.3

A zipped version of the software site can be downloaded here.

Plugin provider: KNIME AG, Zurich, Switzerland
Plugin version: 5.3.0.v202406141453
On NodePit since: 2024-07-09
Last update: 2024-11-15
KNIME versions: Since v3.7

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!