Icon

Spark on hadoop

It is a demo of how to use Spark to perform ML on data on hadoop without livy.Spark processing and building model using data from hadoop. Data file is Wisconsin breastcancer on hadoop file system /user/data/data1.csv Working Dir: /user/data (on hadoop)Host: localhostPort: 9000Working Dir: Current Workflow data areaNo configurationneeded hereFile:/user/data/data1.csvon hadoopTarget: diagnosisNo configneeded hereNode 7Partition HDFS Connector Create Local BigData Environment Table to Spark CSV Reader Spark Decision TreeLearner (MLlib) Spark Predictor(MLlib) Spark Scorer Spark Partitioning It is a demo of how to use Spark to perform ML on data on hadoop without livy.Spark processing and building model using data from hadoop. Data file is Wisconsin breastcancer on hadoop file system /user/data/data1.csv Working Dir: /user/data (on hadoop)Host: localhostPort: 9000Working Dir: Current Workflow data areaNo configurationneeded hereFile:/user/data/data1.csvon hadoopTarget: diagnosisNo configneeded hereNode 7PartitionHDFS Connector Create Local BigData Environment Table to Spark CSV Reader Spark Decision TreeLearner (MLlib) Spark Predictor(MLlib) Spark Scorer Spark Partitioning

Nodes

Extensions

Links