Icon

06_​Modularized_​Spark_​Scripting

Spark Java snippet nodes

This workflow demonstrates the usage of the different Spark Java Snippet nodes to read a text file from HDFS, parse it, filter it and write the result back to HDFS.

You might also want to have a look at the provided snippet templates that each of the node provides. In order to do so simply open the configuration dialog of a Spark Java Snippet node and go to the Templates tab.

Note that this workflow requires that access to a Hadoop cluster running Apache Spark 1.2.1 or newer

Modularize and Execute Your Spark Code Modularized Spark Scripting This workflow demonstrates the usage of the different Spark Java Snippet nodes to read a text file from HDFS, parse it, filter it and write the result back toHDFS. Execute FIRST Execute LAST split into columnsfilter by typefilter invalid rowsrename columnswrite to hdfsRead filefrom HDFSContext is destroyed on closeDelete Demo FileExecute last!!!Connect to HDFSthis node uploads the example file included in the workflow to the hdfs file system. Spark RDDJava Snippet Spark RDDJava Snippet Spark RDDJava Snippet Spark Column Rename Spark RDD JavaSnippet (Sink) Spark to Table Spark RDD JavaSnippet (Source) Create Spark Context(Jobserver) Delete Files HDFS Connection Upload Demo File Modularize and Execute Your Spark Code Modularized Spark Scripting This workflow demonstrates the usage of the different Spark Java Snippet nodes to read a text file from HDFS, parse it, filter it and write the result back toHDFS. Execute FIRST Execute LAST split into columnsfilter by typefilter invalid rowsrename columnswrite to hdfsRead filefrom HDFSContext is destroyed on closeDelete Demo FileExecute last!!!Connect to HDFSthis node uploads the example file included in the workflow to the hdfs file system. Spark RDDJava Snippet Spark RDDJava Snippet Spark RDDJava Snippet Spark Column Rename Spark RDD JavaSnippet (Sink) Spark to Table Spark RDD JavaSnippet (Source) Create Spark Context(Jobserver) Delete Files HDFS Connection Upload Demo File

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: koetter
Created at: 2015-05-27
On NodePit since: 2019-07-12
Last update: 2023-06-07
Created with KNIME version: v4.1.0
Tags: SparkHadoopBig Data