Spark Column Filter

This node allows columns to be filtered from the input Spark DataFrame/RDD while only the remaining columns are passed to the output Spark DataFrame/RDD. Within the dialog, columns can be moved between the Include and Exclude list.

Options

Manual Selection

Include

This list contains the names of those columns in the input Spark DataFrame/RDD to be included in the output Spark DataFrame/RDD.

Exclude

This list contains the names of those columns in the input Spark DataFrame/RDD to be excluded from the output Spark DataFrame/RDD.

Filter

Use one of these fields to filter either the Include or Exclude list for certain column names or name substrings.

Buttons

Use these buttons to move columns between the Include and Exclude list. Single-arrow buttons will move all selected columns. Double-arrow buttons will move all columns (filtering is taken into account).

Enforce Exclusion

Select this option to enforce the current exclusion list to stay the same even if the input Spark DataFrame/RDD specification changes. If some of the excluded columns are not available anymore, a warning is displayed. (New columns will automatically be added to the inclusion list.)

Enforce Inclusion

Select this option to enforce the current inclusion list to stay the same even if the input Spark DataFrame/RDD specification changes. If some of the included columns are not available anymore, a warning is displayed. (New columns will automatically be added to the exclusion list.)

Wildcard/Regex Selection

Type a search pattern which matches columns to move into the Include or Exclude list. Which list is used can be specified. You can use either Wildcards ('?' matching any character, '*' matching a sequence of any characters) or Regex. You can specify whether your pattern should be case sensitive.

Type Selection

Select the column types that you want to include. Column types that are currently not present are depicted in italic.

Views

This node has no views

Workflows

• 01_Fetch_BioAssaysKNIME Hub
• 02 Taxi Demand Prediction on Spark DeploymentKNIME Hub
• 02_Fetch_And_Transform_PubChem_DataKNIME Hub
• 02_Fetch_And_Transform_PubChem_DataKNIME Hub
• 02_Pivot_PubChemDataKNIME Hub
• Show all 99 workflows

Links

• KNIME E-Learning Course: Column Filter

  www.knime.com/knime-introductory-course/chapter3/section1/column-filter
• Java API documentation about regex patterns

  http://download.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html
• Java API documentation about regex matching

  http://download.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html

Developers