CSV to Spark

This Node Is Deprecated — This version of the node has been replaced with a new and improved version. The old version is kept for backwards-compatibility, but for all new workflows we suggest to use the version linked below.
Go to Suggested ReplacementCSV to Spark
Creates a Spark DataFrame/RDD from given CSV file. See CSV Data Source documentation for more information.

Notice: This feature requires at least Apache Spark 1.5.


Upload data source driver or depend on cluster side provided driver.
First line of files will be used to name columns and will not be included in data.
Character used as delimiter between columns (supports escape sequences, e.g. \t or \u0123).
Quote character
Quote character (delimiters inside quotes are ignored).
Escape character
Escape character (escaped quote characters are ignored).
Determines the parsing mode. By default it is PERMISSIVE.
  • PERMISSIVE: tries to parse all lines: nulls are inserted for missing tokens and extra tokens are ignored.
  • DROPMALFORMED: drops lines which have fewer or more tokens than expected or tokens which do not match the schema.
  • FAILFAST: aborts with a RuntimeException if encounters any malformed line.
Valid charset name (see java.nio.charset.Charset).
Automatically infers column types. It requires one extra pass over the data. All types will be assumed string otherwise.
Skip lines beginning with this character.
Null value
Specifies a string that indicates a null value, any fields matching this string will be set as nulls in the DataFrame.
Date format
Specifies a string that indicates the date format to use when reading dates or timestamps. Custom date formats follow the formats at java.text.SimpleDateFormat. This applies to both DateType and TimestampType. By default, it is null which means trying to parse times and date by java.sql.Timestamp.valueOf() and java.sql.Date.valueOf().

Input Ports

Spark compatible connection (HDFS, WebHDFS, HttpFS, S3, Blob Storage, ...)
Optional Spark context. If not connected a context is created based on the settings in the Spark preferences page.

Output Ports

Spark DataFrame/RDD

Popular Predecessors

  • No recommendations found

Popular Successors

  • No recommendations found


This node has no views


  • No workflows found



You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.