Apache Spark™ Documentation

Setup instructions, programming guides, and other documentation are available for each stable version of Spark below:

• Spark 3.5.3
• Spark 3.5.2
• Spark 3.5.1
• Spark 3.5.0
• Spark 3.4.4
• Spark 3.4.3
• Spark 3.4.2
• Spark 3.4.1
• Spark 3.4.0
• Spark 3.3.4
• Spark 3.3.3
• Spark 3.3.2
• Spark 3.3.1
• Spark 3.3.0
• Spark 3.2.4
• Spark 3.2.3
• Spark 3.2.2
• Spark 3.2.1
• Spark 3.2.0
• Spark 3.1.3
• Spark 3.1.2
• Spark 3.1.1
• Spark 3.0.3
• Spark 3.0.2
• Spark 3.0.1
• Spark 3.0.0
• Spark 2.4.8
• Spark 2.4.7
• Spark 2.4.6
• Spark 2.4.5
• Spark 2.4.4
• Spark 2.4.3
• Spark 2.4.2
• Spark 2.4.1
• Spark 2.4.0
• Spark 2.3.4
• Spark 2.3.3
• Spark 2.3.2
• Spark 2.3.1
• Spark 2.3.0
• Spark 2.2.3
• Spark 2.2.2
• Spark 2.2.1
• Spark 2.2.0
• Spark 2.1.3
• Spark 2.1.2
• Spark 2.1.1
• Spark 2.1.0
• Spark 2.0.2
• Spark 2.0.1
• Spark 2.0.0
• Spark 1.6.3
• Spark 1.6.2
• Spark 1.6.1
• Spark 1.6.0
• Spark 1.5.2
• Spark 1.5.1
• Spark 1.5.0
• Spark 1.4.1
• Spark 1.4.0
• Spark 1.3.1
• Spark 1.3.0
• Spark 1.2.1
• Spark 1.1.1
• Spark 1.0.2
• Spark 0.9.2
• Spark 0.8.1
• Spark 0.7.3
• Spark 0.6.2

Documentation for preview releases:

• Spark 4.0.0 preview2
• Spark 4.0.0 preview1
• Spark 3.0.0 preview2
• Spark 3.0.0 preview
• Spark 2.0.0 preview

The documentation linked to above covers getting started with Spark, as well the built-in components MLlib, Spark Streaming, and GraphX.

In addition, this page lists other resources for learning Spark.

Videos

See the Apache Spark YouTube Channel for videos from Spark events. There are separate playlists for videos of different topics. Besides browsing through playlists, you can also find direct links to videos below.

Screencast Tutorial Videos

• Screencast 1: First Steps with Spark
• Screencast 2: Spark Documentation Overview
• Screencast 3: Transformations and Caching
• Screencast 4: A Spark Standalone Job in Scala

Spark Summit Videos

• Videos from Spark Summit 2014, San Francisco, June 30 - July 2 2013
  • Full agenda with links to all videos and slides
  • Training videos and slides
• Videos from Spark Summit 2013, San Francisco, Dec 2-3 2013
  • Full agenda with links to all videos and slides
  • YouTube playlist of all Keynotes
  • YouTube playlist of Track A (Spark Applications)
  • YouTube playlist of Track B (Spark Deployment, Scheduling & Perf, Related projects)
  • YouTube playlist of the Training Day (i.e. the 2nd day of the summit)

Meetup Talk Videos

In addition to the videos listed below, you can also view all slides from Bay Area meetups here.

• Spark 1.0 and Beyond (slides) by Patrick Wendell, at Cisco in San Jose, 2014-04-23
• Adding Native SQL Support to Spark with Catalyst (slides) by Michael Armbrust, at Tagged in SF, 2014-04-08
• SparkR and GraphX (slides: SparkR, GraphX) by Shivaram Venkataraman & Dan Crankshaw, at SkyDeck in Berkeley, 2014-03-25
• Simple deployment w/ SIMR & Advanced Shark Analytics w/ TGFs (slides) by Ali Ghodsi, at Huawei in Santa Clara, 2014-02-05
• Stores, Monoids & Dependency Injection - Abstractions for Spark (slides) by Ryan Weald, at Sharethrough in SF, 2014-01-17
• Distributed Machine Learning using MLbase (slides) by Evan Sparks & Ameet Talwalkar, at Twitter in SF, 2013-08-06
• GraphX Preview: Graph Analysis on Spark by Reynold Xin & Joseph Gonzalez, at Flurry in SF, 2013-07-02
• Deep Dive with Spark Streaming (slides) by Tathagata Das, at Plug and Play in Sunnyvale, 2013-06-17
• Tachyon and Shark update (slides: Shark, Tachyon) by Ali Ghodsi, Haoyuan Li, Reynold Xin, Google Ventures, 2013-05-09
• Spark 0.7: Overview, pySpark, & Streaming by Matei Zaharia, Josh Rosen, Tathagata Das, at Conviva on 2013-02-21
• Introduction to Spark Internals (slides) by Matei Zaharia, at Yahoo in Sunnyvale, 2012-12-18

Training Materials

• Training materials and exercises from Spark Summit 2014 are available online. These include videos and slides of talks as well as exercises you can run on your laptop. Topics include Spark core, tuning and debugging, Spark SQL, Spark Streaming, GraphX and MLlib.
• Spark Summit 2013 included a training session, with slides and videos available on the training day agenda. The session also included exercises that you can walk through on Amazon EC2.
• The UC Berkeley AMPLab regularly hosts training camps on Spark and related projects. Slides, videos and EC2-based exercises from each of these are available online:
  • AMP Camp 4 (Strata Santa Clara, Feb 2014) — focus on BlinkDB, MLlib, GraphX, Tachyon
  • AMP Camp 3 (Berkeley, CA, Aug 2013)
  • AMP Camp 2 (Strata Santa Clara, Feb 2013)
  • AMP Camp 1 (Berkeley, CA, Aug 2012)

Hands-On Exercises

• Hands-on exercises from Spark Summit 2014. These let you install Spark on your laptop and learn basic concepts, Spark SQL, Spark Streaming, GraphX and MLlib.
• Hands-on exercises from Spark Summit 2013. These exercises let you launch a small EC2 cluster, load a dataset, and query it with Spark, Shark, Spark Streaming, and MLlib.

External Tutorials, Blog Posts, and Talks

• Using Spark with MongoDB — by Sampo Niskanen from Wellmo
• Spark Summit 2013 — contained 30 talks about Spark use cases, available as slides and videos
• A Powerful Big Data Trio: Spark, Parquet and Avro — Using Parquet in Spark by Matt Massie
• Real-time Analytics with Cassandra, Spark, and Shark — Presentation by Evan Chan from Ooyala at 2013 Cassandra Summit
• Run Spark and Shark on Amazon Elastic MapReduce — Article by Amazon Elastic MapReduce team member Parviz Deyhim
• Spark, an alternative for fast data analytics — IBM Developer Works article by M. Tim Jones

Books

• Learning Spark, by Holden Karau, Andy Konwinski, Patrick Wendell and Matei Zaharia (O'Reilly Media)
• Spark in Action, by Marko Bonaci and Petar Zecevic (Manning)
• Advanced Analytics with Spark, by Juliet Hougland, Uri Laserson, Sean Owen, Sandy Ryza and Josh Wills (O'Reilly Media)
• Spark GraphX in Action, by Michael Malak (Manning)
• Fast Data Processing with Spark, by Krishna Sankar and Holden Karau (Packt Publishing)
• Machine Learning with Spark, by Nick Pentreath (Packt Publishing)
• Spark Cookbook, by Rishi Yadav (Packt Publishing)
• Apache Spark Graph Processing, by Rindra Ramamonjison (Packt Publishing)
• Mastering Apache Spark, by Mike Frampton (Packt Publishing)
• Big Data Analytics with Spark: A Practitioner's Guide to Using Spark for Large Scale Data Analysis, by Mohammed Guller (Apress)
• Large Scale Machine Learning with Spark, by Md. Rezaul Karim, Md. Mahedi Kaysar (Packt Publishing)
• Big Data Analytics with Spark and Hadoop, by Venkat Ankam (Packt Publishing)

Examples

• The Spark examples page shows the basic API in Scala, Java and Python.

Research Papers

Spark was initially developed as a UC Berkeley research project, and much of the design is documented in papers. The research page lists some of the original motivation and direction.