This node is currently not available in KNIME v5.3 — instead we’re showing this page for KNIME v4.2. You can use the version menu in the title bar to permanently switch your preferred version. This will also show the link to the update site.

CSV to Spark

Creates a Spark DataFrame/RDD from given CSV file. See CSV Data Source documentation for more information.

Notice: This feature requires at least Apache Spark 1.5.

Options

Driver

Upload data source driver or depend on cluster side provided driver.

Header

First line of files will be used to name columns and will not be included in data.

Delimiter

Character used as delimiter between columns (supports escape sequences, e.g. \t or \u0123).

Quote character

Quote character (delimiters inside quotes are ignored).

Escape character

Escape character (escaped quote characters are ignored).

Mode

Determines the parsing mode. By default it is PERMISSIVE.

PERMISSIVE: tries to parse all lines: nulls are inserted for missing tokens and extra tokens are ignored.
DROPMALFORMED: drops lines which have fewer or more tokens than expected or tokens which do not match the schema.
FAILFAST: aborts with a RuntimeException if encounters any malformed line.

Charset

Valid charset name (see java.nio.charset.Charset).

Schema

Automatically infers column types. It requires one extra pass over the data. All types will be assumed string otherwise.

Comments

Skip lines beginning with this character.

Null value

Specifies a string that indicates a null value, any fields matching this string will be set as nulls in the DataFrame.

Date format

Specifies a string that indicates the date format to use when reading dates or timestamps. Custom date formats follow the formats at java.text.SimpleDateFormat. This applies to both DateType and TimestampType. By default, it is null which means trying to parse times and date by java.sql.Timestamp.valueOf() and java.sql.Date.valueOf().

Input Ports

: Spark compatible connection (HDFS, WebHDFS, HttpFS, S3, Blob Storage, ...)
: Required Spark context.

Output Ports

: Spark DataFrame/RDD

Popular Predecessors

HDFS Connection26 %
Create Spark Context19 %
Create Local Big Data Environment16 %
WebHDFS Connection7 %
Create Databricks Environment5 %
Create Local Big Data Environment (Legacy)4 %
HttpFS Connection3 %
SSH Connection3 %
Upload2 %
FTP Connection2 %
Amazon S3 Connection2 %
Text to Spark2 %
Azure Blob Store Connection1 %
HTTP Connection1 %
Java Edit Variable< 1 %
Create Local Big Data Environment< 1 %
KNIME Server Connection< 1 %
Create Spark Context (Livy)< 1 %
Create Spark Context via Livy (preview)< 1 %
File Upload< 1 %
~~Table Column to Variable~~< 1 %
~~Table Row to Variable~~< 1 %
Table Row To Variable Loop Start< 1 %
CSV Writer< 1 %
Databricks File System Connection< 1 %
Secured WebHDFS Connection< 1 %
DB to Spark< 1 %
Avro to Spark< 1 %
CSV to Spark< 1 %
ORC to Spark< 1 %
Table to Spark< 1 %
Create Spark Context (Jobserver)< 1 %
Create Big Data Test Environment (legacy)< 1 %
Amazon S3 File Picker< 1 %
Explorer Browser< 1 %

Popular Successors

Views

This node has no views

Workflows

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.

Installation

To use this node in KNIME, install the extension KNIME Extension for Apache Spark from the below update site following our NodePit Product and Node Installation Guide:

v4.2

A zipped version of the software site can be downloaded here.

Plugin provider: KNIME AG, Zurich, Switzerland

Plugin version: 4.2.0.v202008241922

On NodePit since: 2020-07-15

Last update: 2024-11-12

KNIME versions: From v3.6 to v4.2

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!