Databricks Spark SQL Connector

Options

Database dialect

Choose the registered database dialect here.

Use latest driver version available

If selected, upon execution the node will automatically use the driver with the latest (highest) driver version that is available for the current database type. This has the advantage that you do not need to touch the workflow after a driver update. However, the workflow might break in the rare case that the behavior of the driver e.g. type mapping changes with the newer version.

Database driver

Choose the JDBC driver to connect to the database here. You can select a specific version of the registered drivers via the drop-down list. Additional drivers can be registered via KNIME's preference page "KNIME -> Databases".

Cluster

Select the Databricks compute cluster to connect to. The cluster must be running or will be started automatically when the node is executed.

Terminate compute cluster on disconnect

If selected, the compute cluster will be terminated when the KNIME workflow is closed or the node is reset. This can help reduce costs by ensuring that clusters are not left running when they are no longer needed.

JDBC parameters

This tab allows you to define JDBC driver connection parameter. The value of a parameter can be a constant, variable, credential user, credential password or KNIME URL. The UserAgentEntry parameter is added as default to all Databricks connections to track the usage of KNIME Analytics Platform as Databricks client. If you are not comfortable sharing this information with Databricks you can remove the parameter. However, if you want to promote KNIME as a client with Databricks leave the parameter as is. For more information about the JDBC driver and the UserAgentEntry, refer to the installation and configuration guide which you can find in the docs directory of the driver package.

Input mapping by name

Columns that match the given name (or regular expression) and database type will be mapped to the specified KNIME type.

• Column selection type: The option allows you to select how the column is matched.
  • Manual: Use the exact name of the column
  • Regex: Allow regex expressions to select multiple columns
• Column name: The name or regex expression.
• Database type: SQL type to map from.
• Mapping to: Datatype to map to.

Input mapping by type

Columns that match the given database type will be mapped to the specified KNIME type.

• From type: SQL type to map from.
• To type: Datatype to map to.

Output mapping by name

Columns that match the given name (or regular expression) and KNIME type will be mapped to the specified database type.

• Column selection type: The option allows you to select how the column is matched.
  • Manual: Use the exact name of the column
  • Regex: Allow regex expressions to select multiple columns
• Column name: The name or regex expression.
• Source type: Datatype to map from.
• Mapping to: SQL type to map to.

Output mapping by type

Columns that match the given KNIME type will be mapped to the specified database type.

• From type: Datatype to map from.
• To type: SQL type to map to.

Additional settings

This tab allows you to define KNIME framework properties such as connection handling, advanced SQL dialect settings or logging options. The available properties depend on the selected database type and driver. For more information about the supported parameters see the KNIME Database Extension Guide.

Views

This node has no views

Workflows

• No workflows found

Links

• No links available

Developers