Explore

Big Data Essentials: HDFS, MapReduce and Spark RDD

Approx. 41 hours to complete

Save Course

Go to Course

Course Summary

Learn the essentials of big data and its impact on businesses and society. Get hands-on experience with Hadoop, MapReduce, Spark, and NoSQL databases.

Key Learning Points

Understand the basics of big data and its impact on various industries
Learn popular big data tools such as Hadoop, MapReduce, Spark, and NoSQL databases
Apply big data tools to real-world scenarios and gain hands-on experience

Learning Outcomes

Understand the basics of big data and its impact on various industries
Gain hands-on experience with popular big data tools
Apply big data tools to real-world scenarios and solve business problems

Prerequisites or good to have knowledge before taking this course

Basic programming knowledge
Familiarity with databases

Course Difficulty Level

Intermediate

Course Format

Self-paced
Online
Hands-on

Similar Courses

Big Data Modeling and Management Systems
Big Data Integration and Processing

Related Education Paths

Notable People in This Field

Doug Cutting
Michael Stonebraker

Related Books

Description

Have you ever heard about such technologies as HDFS, MapReduce, Spark? Always wanted to learn these new tools but missed concise starting material? Don’t miss this course either!

In this 6-week course you will: - learn some basic technologies of the modern Big Data landscape, namely: HDFS, MapReduce and Spark; - be guided both through systems internals and their applications; - learn about distributed file systems, why they exist and what function they serve; - grasp the MapReduce framework, a workhorse for many modern Big Data applications; - apply the framework to process texts and solve sample business cases; - learn about Spark, the next-generation computational framework; - build a strong understanding of Spark basic concepts; - develop skills to apply these tools to creating solutions in finance, social networks, telecommunications and many other fields. Your learning experience will be as close to real life as possible with the chance to evaluate your practical assignments on a real cluster. No mocking, a friendly considerate atmosphere to make the process of your learning smooth and enjoyable. Get ready to work with real datasets alongside with real masters! Special thanks to: - Prof. Mikhail Roytberg, APT dept., MIPT, who was the initial reviewer of the project, the supervisor and mentor of half of the BigData team. He was the one, who helped to get this show on the road. - Oleg Sukhoroslov (PhD, Senior Researcher at IITP RAS), who has been teaching MapReduce, Hadoop and friends since 2008. Now he is leading the infrastructure team. - Oleg Ivchenko (PhD student APT dept., MIPT), Pavel Akhtyamov (MSc. student at APT dept., MIPT) and Vladimir Kuznetsov (Assistant at P.G. Demidov Yaroslavl State University), superbrains who have developed and now maintain the infrastructure used for practical assignments in this course. - Asya Roitberg, Eugene Baulin, Marina Sudarikova. These people never sleep to babysit this course day and night, to make your learning experience productive, smooth and exciting.

Outline

Welcome
Why BigData?
Issues BigData can solve
BigData Applications
What is BigData Essentials?
Course Structure
Meet Emeli
Meet Alexey
Meet Ivan
Slack Channel is the quickest way to get answers to your questions

What are BigData and distributed file systems (e.g. HDFS)?
File system exploration
File system managing
File content exploration 1
File content exploration 2
Processes
Scaling Distributed File System
Block and Replica States, Recovery Process 1
Block and Replica States, Recovery Process 2
HDFS Client
Web UI, REST API
Namenode Architecture
Introduction
Text formats
Binary formats 1
Binary formats 2
Compression
How to submit your first assignment
How to Install Docker on Windows 7, 8, 10
Basic Bash Commands
HDFS Lesson Introduction
Gentle Introduction into "curl"
File formats extra (optional)
Grading System: Instructions and Common Problems
Docker Installation Guide
HDFS CLI Playground
Programming Assignment: Instructions and Common Problems
FAQ How to show your code to teaching staff
Slack channel "Bigdata-coursera" - the quickest to solve technical problems.
Distributed File Systems
Big Data and Distributed File Systems

Solving Problems with MapReduce
Unreliable Components 1
Unreliable Components 2
MapReduce
Distributed Shell
Fault Tolerance
Fault Tolerance. Live Demo
Streaming
Streaming in Python
WordCount in Python
Distributed Cache
Environment, Counters
Testing
Combiner
Partitioner
Comparator
Speculative Execution / Backup Tasks
Compression
Hadoop Streaming Assignments: Intro and Code Samples
Hadoop MapReduce Intro
MapReduce Streaming
Hadoop Streaming Final

Solving Problems with MapReduce (practice week)
How to submit your first Hadoop assignment
Hadoop Streaming Assignments: Intro and Code Samples
Hints to Debug Hadoop Streaming Applications
Grading System and Grading System Sandbox User Guide
Programming Assignment: Instructions and Common Problems
Hadoop Streaming Assignments: Instructions
Hint to the "Stop words" programming assignment

Introduction to Apache Spark
Welcome
RDDs
Transformations 1
Transformations 2
Actions
Resiliency
Execution & Scheduling
Caching & Persistence
Broadcast variables
Accumulator variables
Getting started with Spark & Python
Working with text files
Joins
Broadcast & Accumulator variables
Spark UI
Cluster mode
Spark Assignments Intro
Instructions for Spark programming assignment
Lesson 1 Quiz
Lesson 2 Quiz

Introduction to Apache Spark (practice week)
Spark assignments Intro
Building an intuition behind the PMI definition

Real-World Applications
Sampling
Estimating proportions
Means
Medians
Map and Reduce Side Joins
Tabular Data, KeyFieldSelection
Data Skew, Salting
Twitter graph case study
Shortest path
Data and code
Starter for "Reconstructing the path" assignment
Sample estimates
Advanced MapReduce Techniques
Real-World Applications

Summary of User Reviews

Discover the essentials of big data in this comprehensive course on Coursera. Students have given high praise for the course, citing its informative and engaging content. One key aspect that many users appreciated was the practical examples and exercises that helped reinforce their understanding of the material.

Pros from User Reviews

Informative and engaging content
Practical examples and exercises
Great course for beginners
Excellent instructors
Good pace and structure

Cons from User Reviews

Some lectures can be a bit dry
Not enough hands-on coding
Some content can be repetitive
Could benefit from more advanced topics
Limited interaction with other students

Recommended for you

Apache Spark Interview Questions Preparation Course

Learn everything about Apache Spark. Save time in Interview preparation. Apache Spark is one of the fastest growing trend in Data Science and Data engineering world....

Save Course