Spark Missing Value

This node helps handle missing values found in the ingoing Spark data. The first tab in the dialog (labeled Default) provides default handling options for all columns of a given type. These settings apply to all columns in the input table that are not explicitly mentioned in the second tab (labeled Individual). This second tab permits individual settings for each available column (thus, overriding the default). To make use of this second approach, select a column or a list of columns which needs extra handling, click "Add", and set the parameters. Click on the label with the column name(s), will select all covered columns in the column list. To remove this extra handling (and instead use the default handling), click the "Remove" button for this column.

This node requires at least Apache Spark 2.0

Options

Missing Value Handler Selection

Select and configure the missing value handler to be used for data types or columns. Handlers that do not produce valid PMML 4.2 are marked with an asterisk (*).

Fix Value (Double)
Replaces missing values with a double given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (Integer)
Replaces missing values with an integer number given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (Local Date)
Replaces missing values with a local date given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (Local Date Time)
Replaces missing values with a local date time given by the user. This missing value handler produces valid PMML 4.2.

Fix Value
No description provided.

Fix Value (String)
Replaces missing values with a string given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (False)
Replaces missing boolean values with false. This missing value handler produces valid PMML 4.2.

Fix Value (True)
Replaces missing boolean values with true. This missing value handler produces valid PMML 4.2.

Maximum
Finds the column's largest value and replaces all missing values with it. This missing value handler produces valid PMML 4.2.

Mean
Calculates the mean value of all non-missing cells in a column, replaces the missing values with this mean and changes the column type to double. See also Rounded Mean and Truncated Mean. This missing value handler produces valid PMML 4.2.

Median (approximated, target precision 0.001)
Finds the column's approximated median value and replaces all missing values with it. Implementation use space-efficient Online Computation of Quantile Summaries with a relative error of 0.0001. This missing value handler produces valid PMML 4.2.

Median (exact)
Finds the column's exact median value and replaces all missing values with it. For large tables this might be computationally expensive because the table needs to be sorted to find the median. This missing value handler produces valid PMML 4.2.

Minimum
Finds the column's smallest value and replaces all missing values with it. This missing value handler produces valid PMML 4.2.

Most Frequent Value
Calculates the most frequent value in a column and replaces the missing values with it. This missing value handler produces valid PMML 4.2.

Remove Row*
This missing value handler removes rows that have a missing value in the column it is configured for. This missing value handler does not produce standard PMML 4.2!

Rounded Mean
Calculates the mean value of all non-missing cells in a column and replaces the missing values with this (rounded) mean. See also Mean and Truncated Mean. This missing value handler produces valid PMML 4.2.

Input Ports

: Spark data with missing values

Output Ports

: Spark data with replaced values
: PMML documenting the missing value replacements

Popular Predecessors

Spark Column Filter14 %
Spark Category To Number13 %
Hive to Spark10 %
CSV to Spark7 %
Spark Pivot6 %
Show all 32 recommendations

Popular Successors

Views

This node has no views

Workflows

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.

Installation

To use this node in KNIME, install the extension KNIME Extension for Apache Spark (legacy) from the below update site following our NodePit Product and Node Installation Guide:

v5.4

A zipped version of the software site can be downloaded here.

Plugin provider: KNIME AG, Zurich, Switzerland

Plugin version: 5.4.0.v202411151159

On NodePit since: 2024-12-06

Last update: 2024-12-21

KNIME versions: Since v3.6

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!