0 ×

06_​Modularized_​Spark_​Scripting

Workflow

Spark Java snippet nodes

This workflow demonstrates the usage of the different Spark Java Snippet nodes to read a text file from HDFS, parse it, filter it and write the result back to HDFS.

You might also want to have a look at the provided snippet templates that each of the node provides. In order to do so simply open the configuration dialog of a Spark Java Snippet node and go to the Templates tab.

Note that this workflow requires that access to a Hadoop cluster running Apache Spark 1.2.1 or newer

SparkHadoopBig Data
Modularize and Execute Your Spark Code Modularized Spark Scripting This workflow demonstrates the usage of the different Spark Java Snippet nodes to read a text file from HDFS, parse it, filter it and write the result back toHDFS. Execute FIRST Execute LAST split into columnsfilter by typefilter invalid rowsrename columnswrite to hdfsRead filefrom HDFSContext is destroyed on closeDelete Demo FileExecute last!!!Connect to HDFSthis node uploads the example file included in the workflow to the hdfs file system. Spark RDDJava Snippet Spark RDDJava Snippet Spark RDDJava Snippet Spark Column Rename Spark RDD JavaSnippet (Sink) Spark to Table Spark RDD JavaSnippet (Source) Create Spark Context(Jobserver) Delete Files HDFS Connection Upload Demo File Modularize and Execute Your Spark Code Modularized Spark Scripting This workflow demonstrates the usage of the different Spark Java Snippet nodes to read a text file from HDFS, parse it, filter it and write the result back toHDFS. Execute FIRST Execute LAST split into columnsfilter by typefilter invalid rowsrename columnswrite to hdfsRead filefrom HDFSContext is destroyed on closeDelete Demo FileExecute last!!!Connect to HDFSthis node uploads the example file included in the workflow to the hdfs file system. Spark RDDJava Snippet Spark RDDJava Snippet Spark RDDJava Snippet Spark Column Rename Spark RDD JavaSnippet (Sink) Spark to Table Spark RDD JavaSnippet (Source) Create Spark Context(Jobserver) Delete Files HDFS Connection Upload Demo File

Download

Get this workflow from the following link: Download

Nodes

06_​Modularized_​Spark_​Scripting consists of the following 15 nodes(s):

Plugins

06_​Modularized_​Spark_​Scripting contains nodes provided by the following 4 plugin(s):