PySpark for Big Data

Spark Architecture

The course PySpark for Big Data discusses the architecture of Spark, the Spark Cluster Manager and the difference between Batch and Stream Processing.

Hadoop

After a discussion of the Hadoop Distributed File System, parallel operations and working with RDDs, Resilient Distributed Datasets are discussed in the course PySpark for Big Data. The configuration of PySpark applications via SparkConf and SparkContext is also explained.

MapReduce en SQL

Extensive consideration is given to the possible operations on RDDs, including map and reduce. The use of SQL in Spark is also discussed. The GraphX library is discussed and DataFrames is discussed. Iterative algorithms are also treated.

Mlib library

Finally the course PySpark for Big Data pays attention to machine learning with the Mlib library.

Audience PySpark for Big Data

The course PySpark for Big Data is intended for developers and upcoming Data Analysts who want to learn how to use Apache Spark from Python.

Prerequisites training PySpark for Big Data

To participate in this course, some experience with programming is beneficial for understanding. Prior knowledge of Python or big data handling with Apache Spark is not required.

Realization course PySpark for Big Data

The theory is treated on the basis of presentations. Illustrative demos are used to clarify the concepts discussed. There is ample opportunity to practice and alternate theory and practice. The course times are from 9.30 am to 4.30 pm.

Certification course PySpark for Big Data

Participants receive an official certificate PySpark for Big Data after successful completion of the course.

Modules

Module 1 : Python Primer

• Python Syntax
• Python Data Types
• List, Tuples, Dictionaries
• Python Control Flow
• Functions and Parameters
• Modules and Packages
• Comprehensions
• Iterators and Generators
• Python Classes
• Anaconda Environment
• Jupyter Notebooks

Module 2 : Spark Intro

• What is Apache Spark?
• Spark and Python
• PySpark
• Py4j Library
• Data Driven Documents
• RDD's
• Real Time Processing
• Apache Hadoop MapReduce
• Cluster Manager
• Batch versus Stream Processing
• PySpark Shell

Module 3 : HDFS

• Hadoop Environment
• Environment Setup
• Hadoop Stack
• Hadoop Yarn
• Hadoop Distributed File System
• HDFS Architecture
• Parallel Operations
• Working with Partitions
• RDD Partitions
• HDFS Data Locality
• DAG (Direct Acyclic Graph)

Module 4 : SparkConf

• SparkConf Object
• Setting Configuration Properties
• Uploading Files
• SparkContext.addFile
• Logging Configuration
• Storage Levels
• Serialize RDD
• Replicate RDD partitions
• DISK_ONLY
• MEMORY_AND_DISK
• MEMORY_ONLY

Module 5 : SparkContext

• Main Entry Point
• Executor
• Worker Nodes
• LocalFS
• SparkContext Parameters
• Master
• RDD serializer
• batchSize
• Gateway
• JavaSparkContext instance
• Profiler

Module 6 : RDD’s

• Resilient Distributed Datasets
• Key-Value pair RDDs
• Parallel Processing
• Immutability and Fault Tolerance
• Transformation Operations
• Filter, groupBy and Map
• Action Operations
• Caching and persistence
• PySpark RDD Class
• count, collect, foreach,filter
• map, reduce, join, cache

Module 7 : Spark Processing

• SQL support in Spark
• Spark 2.0 Dataframes
• Defining tables
• Importing datasets
• Querying data frames using SQL
• Storage formats
• JSON / Parquet
• GraphX
• GraphX library overview
• GraphX APIs

Module 8 : Broadcast and Accumulator

• Performance Tuning
• Serialization
• Network Traffic
• Disk Persistence
• MarshalSerializer
• Data Type Support
• Python’s Pickle Serializer
• DStreams
• Sliding Window Operations
• Multi Batch and State Operations

Module 9 : Algorithms

• Iterative Algorithms
• Graph Analysis
• Machine Learning API
• mllib.classification
• Random Forest
• Naive Bayes
• Decision Tree
• mllib.clustering
• mllib.linalg
• mllib.regression