Explore

Big Data Integration and Processing

Approx. 18 hours to complete

Save Course

Go to Course

Course Summary

Learn how to integrate and process big data using Hadoop and Spark frameworks with this course. Gain hands-on experience with real-world big data projects and learn from industry experts.

Key Learning Points

Understand the basics of big data integration and processing
Gain hands-on experience with Hadoop and Spark frameworks
Learn from industry experts and real-world big data projects

Learning Outcomes

Understand the basics of big data integration and processing
Gain hands-on experience with Hadoop and Spark
Be able to apply knowledge to real-world big data projects

Prerequisites or good to have knowledge before taking this course

Basic understanding of programming
Familiarity with SQL

Course Difficulty Level

Intermediate

Course Format

Online self-paced course
Video lectures
Hands-on projects

Similar Courses

Big Data Analytics
Data Science Essentials

Related Education Paths

Notable People in This Field

Creator of Hadoop
Creator of Spark

Related Books

Description

At the end of the course, you will be able to:

*Retrieve data from example database and big data management systems *Describe the connections between data management operations and the big data processing patterns needed to utilize them in large-scale analytical applications *Identify when a big data problem needs data integration *Execute simple big data integration and processing on Hadoop and Spark platforms This course is for those new to data science. Completion of Intro to Big Data is recommended. No prior programming experience is needed, although the ability to install applications and utilize a virtual machine is necessary to complete the hands-on assignments. Refer to the specialization technical requirements for complete hardware and software specifications. Hardware Requirements: (A) Quad Core Processor (VT-x or AMD-V support recommended), 64-bit; (B) 8 GB RAM; (C) 20 GB disk free. How to find your hardware information: (Windows): Open System by clicking the Start button, right-clicking Computer, and then clicking Properties; (Mac): Open Overview by clicking on the Apple menu and clicking “About This Mac.” Most computers with 8 GB RAM purchased in the last 3 years will meet the minimum requirements.You will need a high speed internet connection because you will be downloading files up to 4 Gb in size. Software Requirements: This course relies on several open-source software tools, including Apache Hadoop. All required software can be downloaded and installed free of charge (except for data charges from your internet provider). Software requirements include: Windows 7+, Mac OS X 10.10+, Ubuntu 14.04+ or CentOS 6+ VirtualBox 5+.

Outline

Welcome to Big Data Integration and Processing
What is in this Course?
Summary of Big Data Modeling and Management
Why is Big Data Processing Different?
Slides: Summary & Why Is Big Data Processing Different
Downloading and Installing the Cloudera VM Instructions (Windows)
Downloading and Installing the Cloudera VM Instructions (Mac)
Software Installation Frequently Asked Questions (FAQ)
Instructions for Downloading Hands On Datasets
Instructions for Starting Jupyter

Retrieving Big Data (Part 1)
What is Data Retrieval? Part 1
What is Data Retrieval? Part 2
Querying Two Relations
Subqueries
Querying Relational Data with Postgres
Slides: What is Data Retrieval?
Querying Relational Data with Postgres

Retrieving Big Data (Part 2)
Querying JSON Data with MongoDB
Aggregation Functions
Querying Aerospike
Querying Documents in MongoDB
Exploring Pandas DataFrames
Slides: Querying Data Part 2
Querying Documents in MongoDB
Exploring Pandas DataFrames
Retrieving Big Data Quiz
Postgres, MongoDB, and Pandas

Big Data Integration
Overview of Information Integration
A Data Integration Scenario
Integration for Multichannel Customer Analytics
Big Data Management and Processing Using Splunk and Datameer
Why Splunk?
Connected Cars with Ford's OpenXC and Splunk
Big Data Management and Processing using Datameer
Installing Splunk Enterprise on Windows
Installing Splunk Enterprise on Linux
Exploring Splunk Queries
Optional: Creating Pivot Reports in Splunk
Slides: Information Integration
Downloading Splunk Enterprise
Exploring Splunk Queries
Optional: Instructions for Splunk Pivot Tutorial
Information Integration - Quiz
Hands-On With Splunk

Processing Big Data
Big Data Processing Pipelines
Some High-Level Processing Operations in Big Data Pipelines
Aggregation Operations in Big Data Pipelines
Typical Analytical Operations in Big Data Pipelines
Overview of Big Data Processing Systems
The Integration and Processing Layer
Introduction to Apache Spark
Getting Started with Spark
WordCount in Spark
Big Data Processing Pipelines Slides
Big Data Workflow Management
Slides for Big Data Processing Tools and Systems
WordCount in Spark
Pipeline and Tools
WordCount in Spark

Big Data Analytics using Spark
Spark Core: Programming In Spark using RDDs in Pipelines
Spark Core: Transformations
Spark Core: Actions
Spark SQL
Spark Streaming
Spark MLLib
Spark GraphX
Exploring SparkSQL and Spark DataFrames
Analyzing Sensor Data with Spark Streaming
Slides for Module 5 Lesson 1
Slides for Module 5 Lesson 2
Exploring SparkSQL and Spark DataFrames
Instructions for Configuring VirtualBox for Spark Streaming
Analyzing Sensor Data with Spark Streaming
More on Spark
SparkSQL and Spark Streaming

Learn By Doing: Putting MongoDB and Spark to Work
Let's Analyze Soccer Tweets!
Expressing Analytical Questions as MongoDB Queries
Exporting Data from MongoDB to a CSV File
Analyzing Tweets About Countries
Check Your Query Results
Check Your Analysis Results

Summary of User Reviews

Discover the world of big data integration and processing with this highly rated course on Coursera. The course covers various topics related to big data processing and has received positive reviews from many users.

Key Aspect Users Liked About This Course

The course provides practical examples and hands-on assignments that help users understand the concepts better.

Pros from User Reviews

The course provides a comprehensive overview of big data processing and integration.
The instructor is knowledgeable and provides clear explanations.
The hands-on assignments help users apply the concepts learned in the course.
The course is well-structured and easy to follow.
The course provides useful resources and references that users can use to further their understanding of the topic.

Cons from User Reviews

Some users found the course to be too technical and challenging.
The course may not be suitable for beginners who have no prior experience with big data processing.
Some users found the course to be too theoretical and lacking in practical examples.
The course may require a significant time commitment from users.
The course may not provide enough support for users who need additional assistance.

Recommended for you

Complete Data Science Course: Beginner to Advance

Learn Data Science like a professional! Start as a beginner and go all the way to advanced from scratch. Welcome to my course "Complete Data Science Course: Beginner to Advance"....

Save Course

The Data Analyst Course: Complete Data Analyst Bootcamp 2021

Complete Data Analyst Training: Python, NumPy, Pandas, Data Collection, Preprocessing, Data Types, Data Visualization The problem Most data analyst, data science, and coding courses miss a critical practical step....

Save Course

Learning Path: SMACK: Getting Started with the SMACK Stack

Build scalable and efficient data processing platforms If you want to outrun your competitors by taking business decisions using your data, then this course is for you....

Save Course

Securing Investment Returns in the Long Run

You will first learn about absolute and relative performance, risk-adjusted returns and how to decompose investment performance. General Introduction and Key Concepts...

Save Course