0 ×

Spark Missing Value

KNIME Extension for Apache Spark core infrastructure version 4.0.0.v201907300820 by KNIME AG, Zurich, Switzerland

This node helps handle missing values found in the ingoing Spark data. The first tab in the dialog (labeled Default) provides default handling options for all columns of a given type. These settings apply to all columns in the input table that are not explicitly mentioned in the second tab (labeled Individual). This second tab permits individual settings for each available column (thus, overriding the default). To make use of this second approach, select a column or a list of columns which needs extra handling, click "Add", and set the parameters. Click on the label with the column name(s), will select all covered columns in the column list. To remove this extra handling (and instead use the default handling), click the "Remove" button for this column.

This node requires at least Apache Spark 2.0

Options

Missing Value Handler Selection
Select and configure the missing value handler to be used for data types or columns. Handlers that do not produce valid PMML 4.2 are marked with an asterisk (*).

Fix Value (Double)
Replaces missing values with a double given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (Integer)
Replaces missing values with an integer number given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (Local Date)
Replaces missing values with a local date given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (Local Date Time)
Replaces missing values with a local date time given by the user. This missing value handler produces valid PMML 4.2.

Fix Value
No description provided.

Fix Value (String)
Replaces missing values with a string given by the user. This missing value handler produces valid PMML 4.2.

Fix Value (False)
Replaces missing boolean values with false. This missing value handler produces valid PMML 4.2.

Fix Value (True)
Replaces missing boolean values with true. This missing value handler produces valid PMML 4.2.

Maximum
Finds the column's largest value and replaces all missing values with it. This missing value handler produces valid PMML 4.2.

Mean
Calculates the mean value of all non-missing cells in a column, replaces the missing values with this mean and changes the column type to double. See also Rounded Mean and Truncated Mean. This missing value handler produces valid PMML 4.2.

Median (approximated, target precision 0.001)
Finds the column's approximated median value and replaces all missing values with it. Implementation use space-efficient Online Computation of Quantile Summaries with a relative error of 0.0001. This missing value handler produces valid PMML 4.2.

Median (exact)
Finds the column's exact median value and replaces all missing values with it. For large tables this might be computationally expensive because the table needs to be sorted to find the median. This missing value handler produces valid PMML 4.2.

Minimum
Finds the column's smallest value and replaces all missing values with it. This missing value handler produces valid PMML 4.2.

Most Frequent Value
Calculates the most frequent value in a column and replaces the missing values with it. This missing value handler produces valid PMML 4.2.

Remove Row*
This missing value handler removes rows that have a missing value in the column it is configured for. This missing value handler does not produce standard PMML 4.2!

Rounded Mean
Calculates the mean value of all non-missing cells in a column and replaces the missing values with this (rounded) mean. See also Mean and Truncated Mean. This missing value handler produces valid PMML 4.2.

Input Ports

Spark data with missing values

Output Ports

Spark data with replaced values
PMML documenting the missing value replacements

Best Friends (Incoming)

Best Friends (Outgoing)

Workflows

Installation

To use this node in KNIME, install KNIME Extension for Apache Spark from the following update site:

KNIME 4.0
Wait a sec! You want to explore and install nodes even faster? We highly recommend our NodePit for KNIME extension for your KNIME Analytics Platform.

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.