Big Data: Install Spark 2.3.1 on Hadoop 3.0.0 multi-node cluster on Ubuntu

Note: Contact for college projects (real-time) on BigData and Cloud Computing (Amazon Web Services & Google Cloud Platform)

1. Update your OS (Ubuntu 16.04.3 LTS).

sudo apt-get update

2. Install Scala.

Remove older version of Scala, if available.

sudo apt-get remove scala-library scala

Go to Scala website (http://www.scala-lang.org/files/archive/) and download the latest version.

wget http://www.scala-lang.org/files/archive/scala-2.12.6.deb

sudo dpkg -i scala-2.12.6.deb

sudo apt-get install scala

3. Install SBT.

Go to SBT website (http://dl.bintray.com/sbt/debian/) and download it.

wget http://dl.bintray.com/sbt/debian/sbt-1.1.6.deb

sudo dpkg -i sbt-1.1.6.deb

4. If you see error "error: Package curl is not installed.", then please install "curl" package.

sudo apt-get install curl

5. Install SBT again (Run below commands again)

sudo dpkg -i sbt-1.1.6.deb

sudo apt-get update

sudo apt-get install sbt

6. Verify Scala version

scala -version

7. Install Spark

Go to Apache Software Foundation Distribution Directory (https://www.apache.org/dist/) and download it.

wget https://www.apache.org/dist/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz

8. Extract it using Tape Archive command (assuming you're in hadoopuser's home directory)

sudo tar -xzvf spark-2.3.1-bin-hadoop2.7.tgz -C /usr/local/

9. Change owner and mode of installation.

sudo chown -R hadoopuser:hadoopgroup /usr/local/spark-2.3.1-bin-hadoop2.7/

sudo chmod -R 755 /usr/local/spark-2.3.1-bin-hadoop2.7/

10. Update .bashrc file with paths for Spark.

sudo gedit ~/.bashrc

Paste following lines at the end of the file:

export SPARK_HOME=/usr/local/spark-2.3.1-bin-hadoop2.7
export SPARK_CONF_DIR=/usr/local/spark-2.3.1-bin-hadoop2.7/conf
export PATH=$PATH:$SPARK_HOME/bin
export PATH=$PATH:$SPARK_HOME/sbin

11. Copy and remove template extension from files (spark-defaults.conf.template, spark-env.sh.template and slaves.template)

sudo cp /usr/local/spark-2.3.1-bin-hadoop2.7/conf/spark-defaults.conf.template /usr/local/spark-2.3.1-bin-hadoop2.7/conf/spark-defaults.conf

sudo cp /usr/local/spark-2.3.1-bin-hadoop2.7/conf/spark-env.sh.template /usr/local/spark-2.3.1-bin-hadoop2.7/conf/spark-env.sh

sudo cp /usr/local/spark-2.3.1-bin-hadoop2.7/conf/slaves.template /usr/local/spark-2.3.0-bin-hadoop2.7/conf/slaves

12. Edit spark-env.sh and edit below lines:

export HADOOP_CONF_DIR=/usr/local/hadoop-3.0.0/etc/hadoop

export SPARK_MASTER_HOST=ub16043lts00

export SPARK_MASTER_PORT=7077

export SPARK_CONF_DIR=/usr/local/spark-2.3.1-bin-hadoop2.7/conf

13. Edit spark-defaults.conf and edit below lines:

spark.master spark://ub16043lts00:7077
spark.serializer org.apache.spark.serializer.KryoSerializer

14. Edit slaves and add below lines:

#localhost

10.0.1.1
10.0.1.2

Note: Here 10.0.1.1 is our Hadoop master and Hadoop data node and 10.0.1.2 is our Hadoop data node from Hadoop Cluster.

Now we're making 10.0.1.1 here as Spark master node and worker node both and 10.0.1.2 as Spark worker node only.

10.0.1.1 #Hadoop master/data and Spark master/worker
10.0.1.2 #Hadoop data and Spark worker

15. Copy your Spark distribution from master node (10.0.1.1) and paste to your 10.0.1.2 data/worker node at /usr/local/

16. Run Spark Cluster

hadoopuser@ub16043lts00:~$ $SPARK_HOME/sbin/start-all.sh

o/p:

starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.3.1-bin-hadoop2.7/logs/spark-hadoopuser-org.apache.spark.deploy.master.Master-1-ub16043lts00.out

10.0.1.1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.3.1-bin-hadoop2.7/logs/spark-hadoopuser-org.apache.spark.deploy.worker.Worker-1-ub16043lts00.out

10.0.1.2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.3.1-bin-hadoop2.7/logs/spark-hadoopuser-org.apache.spark.deploy.worker.Worker-1-ub16043lts01.out

17. Verify the daemons

hadoopuser@ub16043lts00:~$ jps
8145 NameNode
10163 Jps
8505 SecondaryNameNode
8810 ResourceManager
10091 Worker
8284 DataNode
8941 NodeManager
10015 Master

18. Submit a spark job

On Local:

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[2] /usr/local/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar 100

On Cluster:

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://10.0.1.1:7077 --executor-memory 1G --total-executor-cores 2 /usr/local/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar 100

2017-09-20 14:17:12,524 INFO spark.SparkContext: Running Spark version 2.3.1
2017-09-20 14:17:13,217 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-09-20 14:17:13,513 INFO spark.SparkContext: Submitted application: Spark Pi
2017-09-20 14:17:13,673 INFO spark.SecurityManager: Changing view acls to: hadoopuser
2017-09-20 14:17:13,674 INFO spark.SecurityManager: Changing modify acls to: hadoopuser
2017-09-20 14:17:13,674 INFO spark.SecurityManager: Changing view acls groups to:
2017-09-20 14:17:13,675 INFO spark.SecurityManager: Changing modify acls groups to:
2017-09-20 14:17:13,676 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoopuser); groups with view permissions: Set(); users with modify permissions: Set(hadoopuser); groups with modify permissions: Set()
2017-09-20 14:17:14,473 INFO util.Utils: Successfully started service 'sparkDriver' on port 42097.
2017-09-20 14:17:14,570 INFO spark.SparkEnv: Registering MapOutputTracker
2017-09-20 14:17:14,638 INFO spark.SparkEnv: Registering BlockManagerMaster
2017-09-20 14:17:14,651 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2017-09-20 14:17:14,652 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
2017-09-20 14:17:14,665 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-09342467-05aa-41ff-82b6-e430adebe197
2017-09-20 14:17:14,725 INFO memory.MemoryStore: MemoryStore started with capacity 413.9 MB
2017-09-20 14:17:14,865 INFO spark.SparkEnv: Registering OutputCommitCoordinator
2017-09-20 14:17:15,058 INFO util.log: Logging initialized @6919ms
2017-09-20 14:17:15,206 INFO server.Server: jetty-9.3.z-SNAPSHOT
2017-09-20 14:17:15,231 INFO server.Server: Started @7094ms
2017-09-20 14:17:15,287 INFO server.AbstractConnector: Started ServerConnector@3047c2e1{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2017-09-20 14:17:15,287 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
2017-09-20 14:17:15,337 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@726386ed{/jobs,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,338 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7249dadf{/jobs/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,338 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@66238be2{/jobs/job,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,344 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@750fe12e{/jobs/job/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,345 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3e587920{/stages,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,346 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@24f43aa3{/stages/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,348 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1e11bc55{/stages/stage,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,351 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6ab72419{/stages/stage/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,353 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4fdfa676{/stages/pool,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,355 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5be82d43{/stages/pool/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,363 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@345e5a17{/storage,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,364 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@443dbe42{/storage/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,365 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1734f68{/storage/rdd,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,366 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ed190be{/storage/rdd/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,367 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5bbc9f97{/environment,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,368 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@41382722{/environment/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,369 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@425357dd{/executors,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,370 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@210386e0{/executors/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,371 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@65f87a2c{/executors/threadDump,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,372 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6ce1f601{/executors/threadDump/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,386 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1e886a5b{/static,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,389 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1edb61b1{/,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,392 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@cc62a3b{/api,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,397 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5305c37d{/jobs/job/kill,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,398 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3dddbe65{/stages/stage/kill,null,AVAILABLE,@Spark}
2017-09-20 14:17:15,401 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.1.1:4040
2017-09-20 14:17:15,450 INFO spark.SparkContext: Added JAR file:/usr/local/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar at spark://10.0.1.1:42097/jars/spark-examples_2.11-2.3.0.jar with timestamp 1505897235449
2017-09-20 14:17:15,860 INFO client.StandaloneAppClient$ClientEndpoint: Connecting to master spark://10.0.1.1:7077...
2017-09-20 14:17:16,027 INFO client.TransportClientFactory: Successfully created connection to /10.0.1.1:7077 after 107 ms (0 ms spent in bootstraps)
2017-09-20 14:17:16,377 INFO cluster.StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20170920141716-0000
2017-09-20 14:17:16,399 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33869.
2017-09-20 14:17:16,407 INFO netty.NettyBlockTransferService: Server created on 10.0.1.1:33869
2017-09-20 14:17:16,423 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2017-09-20 14:17:16,427 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.1.1, 33869, None)
2017-09-20 14:17:16,460 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.0.1.1:33869 with 413.9 MB RAM, BlockManagerId(driver, 10.0.1.1, 33869, None)
2017-09-20 14:17:16,484 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.1.1, 33869, None)
2017-09-20 14:17:16,500 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.0.1.1, 33869, None)
2017-09-20 14:17:16,537 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20170920141716-0000/0 on worker-20170920133953-10.0.1.1-38285 (10.0.1.1:38285) with 1 cores
2017-09-20 14:17:16,546 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20170920141716-0000/0 on hostPort 10.0.1.1:38285 with 1 cores, 1024.0 MB RAM
2017-09-20 14:17:16,548 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20170920141716-0000/1 on worker-20170920133946-10.0.1.2-42499 (10.0.1.2:42499) with 1 cores
2017-09-20 14:17:16,551 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20170920141716-0000/1 on hostPort 10.0.1.2:42499 with 1 cores, 1024.0 MB RAM
2017-09-20 14:17:16,960 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20170920141716-0000/1 is now RUNNING
2017-09-20 14:17:17,015 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20170920141716-0000/0 is now RUNNING
2017-09-20 14:17:17,363 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@a619c2{/metrics/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:17,470 INFO cluster.StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
2017-09-20 14:17:17,896 INFO internal.SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/hadoopuser/spark-warehouse/').
2017-09-20 14:17:17,897 INFO internal.SharedState: Warehouse path is 'file:/home/hadoopuser/spark-warehouse/'.
2017-09-20 14:17:17,920 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@102d92c4{/SQL,null,AVAILABLE,@Spark}
2017-09-20 14:17:17,926 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60723d6a{/SQL/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:17,941 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@abbc908{/SQL/execution,null,AVAILABLE,@Spark}
2017-09-20 14:17:17,942 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@9a7a808{/SQL/execution/json,null,AVAILABLE,@Spark}
2017-09-20 14:17:17,945 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4650a407{/static/sql,null,AVAILABLE,@Spark}
2017-09-20 14:17:23,981 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.0.1.2:41024) with ID 1
2017-09-20 14:17:24,500 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.0.1.2:43311 with 366.3 MB RAM, BlockManagerId(1, 10.0.1.2, 43311, None)
2017-09-20 14:17:26,531 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
2017-09-20 14:17:31,120 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:38
2017-09-20 14:17:31,204 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.0.1.1:44202) with ID 0
2017-09-20 14:17:31,801 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 100 output partitions
2017-09-20 14:17:31,803 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2017-09-20 14:17:31,866 INFO scheduler.DAGScheduler: Parents of final stage: List()
2017-09-20 14:17:31,917 INFO scheduler.DAGScheduler: Missing parents: List()
2017-09-20 14:17:32,148 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.0.1.1:39667 with 413.9 MB RAM, BlockManagerId(0, 10.0.1.1, 39667, None)
2017-09-20 14:17:32,189 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2017-09-20 14:17:35,212 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)
2017-09-20 14:17:36,221 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1172.0 B, free 413.9 MB)
2017-09-20 14:17:36,302 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.1.1:33869 (size: 1172.0 B, free: 413.9 MB)
2017-09-20 14:17:36,505 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
2017-09-20 14:17:36,717 INFO scheduler.DAGScheduler: Submitting 100 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
2017-09-20 14:17:36,719 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 100 tasks
2017-09-20 14:17:37,473 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 10.0.1.1, executor 0, partition 0, PROCESS_LOCAL, 4829 bytes)
2017-09-20 14:17:37,499 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 10.0.1.2, executor 1, partition 1, PROCESS_LOCAL, 4829 bytes)
2017-09-20 14:17:40,241 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.1.2:43311 (size: 1172.0 B, free: 366.3 MB)
2017-09-20 14:17:41,378 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.1.1:39667 (size: 1172.0 B, free: 413.9 MB)
2017-09-20 14:17:44,483 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 10.0.1.1, executor 0, partition 2, PROCESS_LOCAL, 4829 bytes)
2017-09-20 14:17:44,569 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 7243 ms on 10.0.1.1 (executor 0) 2017-09-20 14:17:57,408 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 20.367 s
2017-09-20 14:17:57,408 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
2017-09-20 14:17:57,519 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 26.397268 s
Pi is roughly 3.1420691142069113
2017-09-20 14:17:57,879 INFO server.AbstractConnector: Stopped Spark@3047c2e1{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2017-09-20 14:17:57,899 INFO ui.SparkUI: Stopped Spark web UI at http://10.0.1.1:4040
2017-09-20 14:17:57,987 INFO cluster.StandaloneSchedulerBackend: Shutting down all executors
2017-09-20 14:17:57,990 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
2017-09-20 14:17:58,409 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
2017-09-20 14:17:58,825 INFO memory.MemoryStore: MemoryStore cleared
2017-09-20 14:17:58,826 INFO storage.BlockManager: BlockManager stopped
2017-09-20 14:17:58,827 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
2017-09-20 14:17:59,486 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
2017-09-20 14:17:59,648 INFO spark.SparkContext: Successfully stopped SparkContext
2017-09-20 14:17:59,664 INFO util.ShutdownHookManager: Shutdown hook called
2017-09-20 14:17:59,666 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-15ed46de-02ae-46dc-936b-5db96c684192

19. Spark UI