CSV to Spark

Options

Settings

Mode

Select if you like to read a file or folder.

File/Folder

Enter the input path. The required syntax of a path depends on the connected file system. The node description of the respective connector node describes the required path format. You can also choose a previously selected file from the drop-down list, or select a location from the "Browse..." dialog. Note that browsing is disabled in some cases:

• Browsing is disabled if the connector node hasn't been executed since the workflow has been opened. (Re)execute the connector node to enable browsing.

The location can be exposed as or automatically set via a path flow variable.

Driver

Upload data source driver or depend on cluster side provided driver.

Header

First line of files will be used to name columns and will not be included in data.

Delimiter

Character used as delimiter between columns (supports escape sequences, e.g. \t or \u0123).

Quote character

Quote character (delimiters inside quotes are ignored).

Escape character

Escape character (escaped quote characters are ignored).

Mode

Determines the parsing mode. By default it is PERMISSIVE.

• PERMISSIVE: tries to parse all lines: nulls are inserted for missing tokens and extra tokens are ignored.
• DROPMALFORMED: drops lines which have fewer or more tokens than expected or tokens which do not match the schema.
• FAILFAST: aborts with a RuntimeException if encounters any malformed line.

Charset

Valid charset name (see java.nio.charset.Charset).

Schema

Automatically infers column types. It requires one extra pass over the data. All types will be assumed string otherwise.

Comments

Skip lines beginning with this character.

Null value

Specifies a string that indicates a null value, any fields matching this string will be set as nulls in the DataFrame.

Date format

Specifies a string that indicates the date format to use when reading dates or timestamps. Custom date formats follow the formats at java.text.SimpleDateFormat. This applies to both DateType and TimestampType. By default, it is null which means trying to parse times and date by java.sql.Timestamp.valueOf() and java.sql.Date.valueOf().

Views

This node has no views

Workflows

• 01_Fetch_BioAssaysKNIME Hub
• 02_Pivot_PubChemDataKNIME Hub
• 06_Connecting_to_DatabricksKNIME Hub
• 07_Will_They_Blend_BigQuery_DatabricksKNIME Hub
• 10_Recommendation_Engine_w_Spark_Collaborative_FilteringKNIME Hub
• CSV_Processing_with_Big_Data_Extension-or-DuckDBKNIME Hub
• kn_forum_62280_spark_excel_exportKNIME Hub
• Tool Migration - From Excel to Value with KNIMEKNIME Hub

Links

• No links available

Developers