Explore

Managing Big Data in Clusters and Cloud Storage

Approx. 21 hours to complete

Save Course

Go to Course

Course Summary

Learn how to store and analyze big data with cloud storage and SQL in this course. Gain valuable skills to help you tackle big data analysis in various industries.

Key Learning Points

Understand the basics of cloud storage and big data analysis
Learn how to use SQL to manipulate and analyze data
Gain practical skills through hands-on projects

Learning Outcomes

Ability to store and analyze big data with cloud storage and SQL
Practical skills through hands-on projects
Understanding of data manipulation techniques

Prerequisites or good to have knowledge before taking this course

Basic understanding of programming concepts
Familiarity with SQL

Course Difficulty Level

Intermediate

Course Format

Online
Self-paced

Similar Courses

Data Warehousing for Business Intelligence
Big Data Essentials: HDFS, MapReduce, and Spark RDD
Introduction to Data Analysis Using Excel

Related Education Paths

Notable People in This Field

Data Scientist

Related Books

Description

In this course, you'll learn how to manage big datasets, how to load them into clusters and cloud storage, and how to apply structure to the data so that you can run queries on it using distributed SQL engines like Apache Hive and Apache Impala. You’ll learn how to choose the right data types, storage systems, and file formats based on which tools you’ll use and what performance you need.

By the end of the course, you will be able to • use different tools to browse existing databases and tables in big data systems; • use different tools to explore files in distributed big data filesystems and cloud storage; • create and manage big data databases and tables using Apache Hive and Apache Impala; and • describe and choose among different data types and file formats for big data systems. To use the hands-on environment for this course, you need to download and install a virtual machine and the software on which to run it. Before continuing, be sure that you have access to a computer that meets the following hardware and software requirements: • Windows, macOS, or Linux operating system (iPads and Android tablets will not work) • 64-bit operating system (32-bit operating systems will not work) • 8 GB RAM or more • 25GB free disk space or more • Intel VT-x or AMD-V virtualization support enabled (on Mac computers with Intel processors, this is always enabled; on Windows and Linux computers, you might need to enable it in the BIOS) • For Windows XP computers only: You must have an unzip utility such as 7-Zip or WinZip installed (Windows XP’s built-in unzip utility will not work)

Knowledge

Use different tools to browse existing databases and tables in big data systems
Use different tools to explore files in distributed big data filesystems and cloud storage
Create and manage big data databases and tables using Apache Hive and Apache Impala
Describe and choose among different data types and file formats for big data systems

Outline

Orientation to Data in Clusters and Cloud Storage
Welcome to the Course
Browsing Tables with Hue
Browsing Tables with SQL Utility Statements
Browsing HDFS with the Hue File Browser
Browsing HDFS from the Command Line
Understanding S3 and Other Cloud Storage Platforms
Browsing S3 Buckets from the Command Line
Review and Preparation
Instructions for Downloading and Installing the Exercise Environment
Troubleshooting the VM
Week 1 Graded Quiz

Defining Databases, Tables, and Columns
Week 2 Introduction
Introduction to the CREATE TABLE Statement
Using Different Schemas on the Same Data
Specifying TBLPROPERTIES
Examining, Modifying, and Removing Tables
Hive and Impala Interoperability
Impala Metadata Refresh
Creating Databases and Tables with Hue
Creating Databases and Tables with SQL
Permissions to Create Databases and Tables
The ROW FORMAT Clause
The STORED AS Clause
The LOCATION Clause
CREATE TABLE Shortcuts
Using Hive SerDes
Working with Unstructured and Semi-Structured Data
Examining Table Structure
Dropping Databases and Tables
Modifying Existing Tables
Week 2 Practice Quiz
Week 2 Graded Quiz

Data Types and File Types
Week 3 Introduction
Overview of Data Types
Choosing the Right Data Types
Overview of File Types
Choosing the Right File Types
Integer Data Types
Decimal Data Types
Character String Data Types
Other Data Types
Examining Data Types
Out-of-Range Values
Text Files
Avro Files
Parquet Files
ORC Files
Other File Types
Creating Tables with Avro and Parquet Files
Week 3 Practice Quiz
Week 3 Graded Quiz

Managing Datasets in Clusters and Cloud Storage
Week 4 Introduction
Refresh Impala's Metadata Cache after Loading Data
Loading Files into HDFS with Hue's Table Browser
Loading Files into HDFS with Hue's File Browser
Loading Files into HDFS from the Command Line
Loading Files into S3 from the Command Line
Using Hive and Impala to Load Data into Tables
Conclusion
More about HDFS Shell Commands
Chaining and Scripting with HDFS Commands
HDFS Permissions
Other Ways to Load Files into S3
S3 Permissions
Missing Values
Character Sets
Using Sqoop to Import Data
More Sqoop Import Options
Using Sqoop to Export Data
SQL LOAD DATA Statements
SQL INSERT Statements
SQL INSERT ... SELECT and CTAS Statements
Week 4 Practice Quiz
Week 4 Graded Quiz

Optimizing Hive and Impala (Honors)
Week 5 Introduction
What to Do When Queries Are Too Complex
What to Do When Queries Take Too Long
When to Use Table Partitioning
When to Use Complex Columns
File Systems versus Storage Engines
Creating and Querying Views
Modifying and Removing Views
Materialized and Non-Materialized Views
The ORDER BY Clause in Views
Choosing Which Query Engine to Use
Understanding Map Tasks and Reduce Tasks
Hive Query Performance Patterns
Understanding Execution Plans
Table and Column Statistics
Other Strategies for Query Optimization
Creating Partitioned Tables
Loading Data with Dynamic Partition
Loading Data with Static Partitioning
Risks of Using Partitioning
Complex Data Types
Creating Tables with Complex Data
Querying Complex Data with Hive
Querying Complex Data with Impala
Complex Data in Practice
Overview of Apache Kudu
Week 5 Practice Quiz
Week 5 Graded Quiz

Summary of User Reviews

Read reviews about the Cloud Storage & Big Data Analysis with SQL course on Coursera. It has received positive reviews overall. Many users appreciated the practical examples provided throughout the course.

Key Aspect Users Liked About This Course

Practical examples provided throughout the course.

Pros from User Reviews

Well-structured course content
Easy to follow explanations
Great introduction to cloud storage and big data analysis
Good for beginners

Cons from User Reviews

Lack of in-depth analysis
Not suitable for advanced learners
Some technical issues reported
Course materials not always up-to-date

Recommended for you

Elasticsearch 7 and the Elastic Stack: In Depth and Hands On

Complete Elasticsearch tutorial - search, analyze, and visualize big data with Elasticsearch, Kibana, Logstash, & Beats. This course covers it all, from installation to operations, with over 100 lectures including 11 hours of video....

Save Course

AWS Data Architect Bootcamp - 43 Services 500 FAQs 20+ Tools

AWS Databases, EMR, SageMaker, IoT, Redshift, Glue, QuickSight, RDS, Aurora, DynamoDB, Kinesis, Rekognition & much more This is the most-comprehensive AWS course related to AWS data architecture on the market....

Save Course

OpenStack: Design and Implement Cloud Infrastructure

Design and implement custom clouds with OpenStack OpenStack is an open source cloud computing platform that is rapidly gaining popularity in enterprise data centres....

Save Course

Security Best Practices in Google Cloud

This self-paced training course gives participants broad study of security controls and techniques on Google Cloud. It also covers securing Kubernetes environments....

Save Course