Apache Spark™ is a unified analytics engine for large-scale data processing, and one of the first steps toward big data analytics. So let's get started.
Most current cluster programming models are based on acyclic data flow — from stable storage to stable storage. For instance, Hadoop reads data from persistent storage in the Map step, then writes it back to persistent storage (HDFS) in Reduce. This is convenient for dynamically assigning machines or handling failures.
The architecture holds up until we need to iterate over the data — iteration is very expensive in most clustering models. That's where Apache Spark comes to the rescue.
Spark can be 100x faster than Hadoop for large-scale data processing by exploiting in-memory computing and other optimizations. Spark is also fast when data is stored on disk, and currently holds the world record for large-scale on-disk sorting.
Now let's get our hands dirty. The prerequisites for this installation guide are:
- An Ubuntu system (we developers love open source more than anything)
- Access to a terminal or command line (
ctrl + alt + T) - A user with sudo or root permissions (sudo — never forget your roots)
Apache Spark has a few dependencies, and we need to get them before we get started:
- JDK
- Scala
- Git
One perk of using Linux: we can install all these dependencies at once.
sudo apt install default-jdk scala git -y
The installation can be verified with:
java -version; javac -version; scala -version; git --version
Now it's time to download the MVP, Apache Spark. We'll use wget and the direct link to the package (check for the latest version at downloads.apache.org/spark):
wget https://downloads.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz
Let's unzip the package and move the folder to /opt/spark:
tar xvf spark-3.0.1-bin-hadoop2.7.tgz
sudo mv spark-3.0.1-bin-hadoop2.7 /opt/spark
Configuring the environment is one of the trickiest parts, so be careful about typos and directories. We need to add a few lines to the user profile, generally found at ~/.profile. One way to add the lines is by running echo, as shown:
echo "export SPARK_HOME=/opt/spark" >> ~/.profile
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.profile
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.profile
Reload the user profile by running source ~/.profile.
Let's test the Spark shell:
spark-shell
The shell shown is specific to Scala. If you want, you can always run the Spark shell in Python instead — quit Scala by typing :q, then enter the Python Spark shell with the command pyspark.
Congratulations! Your Spark installation is up and running. Now, before we leave, a few basic commands.
To start a master server instance on the current machine:
start-master.sh
In this single-server, standalone setup, to start one slave server alongside the master server, run the following (remember to swap in your hostname:port):
start-slave.sh spark://master:port
To stop the master instance:
stop-master.sh
To stop a running worker process:
stop-slave.sh
There's a lot more to learn and experiment with, but until next time!
Citations
- pesitsouth.pes.edu — study material
- Apache Spark™: What is Spark — databricks.com
- How to Install Spark on Ubuntu — phoenixnap.com