Explore

Custom and Distributed Training with TensorFlow

Approx. 24 hours to complete

Save Course

Go to Course

Course Summary

Learn how to customize and distribute TensorFlow training on multiple machines with this course. Explore the architecture of distributed training, use TensorFlow's High-Level API to train models, and scale training to multiple GPUs and machines.

Key Learning Points

Understand distributed training with TensorFlow
Learn how to customize TensorFlow training to your needs
Scale training to multiple GPUs and machines

Learning Outcomes

Understand the architecture of distributed training
Train models using TensorFlow's High-Level API
Scale training to multiple GPUs and machines

Prerequisites or good to have knowledge before taking this course

Familiarity with machine learning concepts
Basic knowledge of TensorFlow

Course Difficulty Level

Intermediate

Course Format

Online
Self-paced

Similar Courses

Advanced Machine Learning with TensorFlow on Google Cloud Platform
Applied Data Science with Python

Related Education Paths

Notable People in This Field

Andrew Ng
Ian Goodfellow

Related Books

Description

In this course, you will:

• Learn about Tensor objects, the fundamental building blocks of TensorFlow, understand the difference between the eager and graph modes in TensorFlow, and learn how to use a TensorFlow tool to calculate gradients. • Build your own custom training loops using GradientTape and TensorFlow Datasets to gain more flexibility and visibility with your model training. • Learn about the benefits of generating code that runs in graph mode, take a peek at what graph code looks like, and practice generating this more efficient code automatically with TensorFlow’s tools. • Harness the power of distributed training to process more data and train larger models, faster, get an overview of various distributed training strategies, and practice working with a strategy that trains on multiple GPU cores, and another that trains on multiple TPU cores. The DeepLearning.AI TensorFlow: Advanced Techniques Specialization introduces the features of TensorFlow that provide learners with more control over their model architecture and tools that help them create and train advanced ML models. This Specialization is for early and mid-career software and machine learning engineers with a foundational understanding of TensorFlow who are looking to expand their knowledge and skill set by learning advanced TensorFlow features to build powerful models.

Outline

Differentiation and Gradients
A conversation with Andrew Ng: Overview of course 2
What is a tensor?
Creating tensors in code
Math operations with tensors
Basic Tensors code walkthrough
Broadcasting, operator overloading and Numpy compatibility
Evaluating variables and changing data types
Gradient Tape
Gradient Descent using Gradient Tape
Calculate gradients on higher order functions
Persistent=true and higher order gradients
Gradient Tape basics code walkthrough
Connect with your mentors and fellow learners on Slack!
Reference: CNN for visual recognition
Tensors and Gradient Tape

Custom Training
Custom Training Loop steps
Loss and gradient descent
Define Training Loop and Validate Model
Training Basics code walkthrough
Training steps and data pipeline
Define the training loop
Gradients, metrics, and validation
Fashion MNIST Custom Training Loop code walkthrough
Reference: tf.keras.metrics
Custom Training

Graph Mode
Benefits of graph mode
Generating graph code
AutoGraph Basics code walkthrough
Control dependencies and flows
Loops and tracing variables
AutoGraph code walkthrough
Reference: Fizz Buzz
AutoGraph

Distributed Training
Intro to distribution strategies
Types of distribution strategies
Converting code to the Mirrored Strategy
Mirrored Strategy code walkthrough
Custom Training for Multiple GPU Mirrored Strategy
Multi GPU Mirrored Strategy code walkthrough
TPU Strategy
TPU Strategy code walkthrough
Other Distributed Strategies
References used in Other Distributed Strategies
References
Acknowledgments
Distributed Strategy

Summary of User Reviews

This course on custom distributed training with TensorFlow received high praise from many users for its comprehensive coverage and practical examples. Users appreciated the instructor's expertise and clear explanations.

Key Aspect Users Liked About This Course

The practical examples provided in the course were a standout feature for many users, allowing them to apply the concepts to real-world scenarios.

Pros from User Reviews

Comprehensive coverage of custom distributed training with TensorFlow
Practical examples that help users apply the concepts in real-world scenarios
Expert instructor with clear explanations
Engaging assignments that challenge users to hone their skills
Great community support and resources

Cons from User Reviews

Some users found the course content to be too advanced for beginners
The course assumes a certain level of familiarity with TensorFlow
Some users found the pace of the course to be too fast
Lack of hands-on coding exercises in some sections
Some users felt that the course could benefit from more interactive elements

Recommended for you

Tensorflow 2.0: Deep Learning and Artificial Intelligence

Neural Networks for Computer Vision, Time Series Forecasting, NLP, GANs, Reinforcement Learning, and More! Welcome to Tensorflow 2. 0! What an exciting time....

Save Course

R: Neural Nets and CNN Architecture in R - Masterclass!

One stop guide to implement cutting-edge CNN architectures and build Neural Network models to solve complex problems! You’ll also apply supervised and unsupervised learning to your daily projects....

Save Course

Learn Data Science Deep Learning, Machine Learning NLP & R

Learn Data Science, Deep Learning, Machine Learning, Natural Language Processing, R and Python Language with libraries || DATA SCIENCE || What Does a Data Scientist Do?...

Save Course

ML Pipelines on Google Cloud

In this course, you will be learning from ML Engineers and Trainers who work with the state-of-the-art development of ML pipelines here at Google Cloud....

Save Course