Reinforcement learning (RL) is a hugely popular area of deep learning, and many data scientists are exploring this AI technology to broaden their skillet to include a number of important problem domains like chatbots, robotics, discrete optimization, web automation and much more. As a result of this wide-spread interest in RL, there are many available educational resources specifically tailored to this class of deep learning – boot camps, training certificates, educational specializations, etc. But if you’re a data scientist who has been programming in Python (with object oriented features) for a while, and has some experience with other forms of deep learning using a framework like TensorFlow, then maybe this new book, “Deep Reinforcement Learning Hands-On,” by Maxim Lapan from Packt, might be a great way to kick-start yourself into becoming productive with RL.
Reinforcement Learning
RL development is being driven by a number of large companies and research groups, including Google, Microsoft, and Facebook. RL requires considerable investment in research as the field is growing to enable data scientists to be able to take prescribed methods and apply them to a problem domain. This is similar to how the states of computer vision and NLP were several years ago. A good portion of the AI research papers appearing on the arXiv.org pre-print server deal with RL.
As a journalist, I’m seeing that the field of RL is attracting lots of attention, both from researchers and practitioners. Fortunately, this hands-on book helps readers to understand RL methods using real-life problems, and make the exciting RL field accessible to a much wider audience than just research groups or large AI companies.
The emphasis of the book is to enable readers to feel that modern RL is an approachable, practical, and useful aspect of machine learning. The book will help readers understand RL methods using real-life, relatable problems. Information about recent research is widely available via arXiv.org and research blogs, but it is too specialized and grounded with mathematics for many data scientists to easily consume. If readers are new to this field, these papers might not be the best source of information, at least not early on. This book gets around such complexities by addressing the lack of practical and structured information about RL methods and approaches.
New in the Second Edition
The second edition is devoted to the very latest RL tools and techniques, focusing on new innovations in this emerging field. It includes six new chapters that give readers the hands-on ability to code intelligent learning agents to perform a range of practical tasks. New topics include robotics and multiple agent learning. The chapter on trust regions has been updated with the new SAC algorithm. The author has also developed a library for easily interfacing the OpenAI environment which was used all through the first edition but is now properly documented.
After a clear definition of RL and associated nomenclature, the Gym library and its classes are explained. Gym is a rich library used widely by RL practitioners and the book’s code depends on it. A straight-forward and impressive RL application using Gym is given and explained carefully.
Fun topics include – using the TextWorld environment from Microsoft Research to solve text-based interactive fiction games, solving discrete optimization problems (showcased using Rubik’s Cube), and learning to apply RL methods to the robotics domain.
The book then dives into an important aspect of RL methods: advanced exploration. Several modern exploration techniques are described, implemented, and compared. Readers will also discover basic methods applied to a multi-agent environment.
All examples have been updated for PyTorch 1.3. A good introduction to PyTorch is given in Chapter 3. PyTorch Ignite has also been introduced to make the RL coding more concise.
To demonstrate the breadth of coverage of the subject, here are the chapters included in the book:
- Chapter 1 – What is Reinforcement Learning?
- Chapter 2 – OpenAI Gym
- Chapter 3 – Deep Learning with PyTorch
- Chapter 4 – The Cross-Entropy Method
- Chapter 5 – Tabular Learning and the Bellman Equation
- Chapter 6 – Deep Q-Networks
- Chapter 7 – Higher-Level RL Libraries
- Chapter 8 – DQN Extensions
- Chapter 9 – Ways to Speed up RL Methods
- Chapter 10 – Stocks Trading Using RL
- Chapter 11 – Policy Gradients
- Chapter 12 – The Actor-Critic Method
- Chapter 13 – Asynchronous Advantage Actor-Critic
- Chapter 14 – Training Chatbots with RL
- Chapter 15 – The TextWorld Environment
- Chapter 16 – Web Navigation
- Chapter 17 – Continuous Action Space
- Chapter 18 – RL in Robotics
- Chapter 19 – Trust Regions – TRPO, PPO, and ACKTR
- Chapter 20 – Black-Box Optimization in RL
- Chapter 21 – Advanced Exploration
- Chapter 22 – Beyond Model-Free
- Chapter 23 – AlphaGo Zero
- Chapter 24 – RL in Discrete Optimization
- Chapter 25 – Multi-agent RL
The book is targeted toward readers with a fluency in Python. Basic deep learning approaches should be familiar to readers and some practical experience in DL will be helpful. This book is a complete introduction to deep reinforcement learning and requires no background in RL.
This is a very comprehensive book covering a range of RL techniques. With nearly 800 pages, it will also weigh down your bookshelf a bit. It comes with a GitHub repo containing all the code described in the book. Great place to start building your RL skillet.
Contributed by Daniel D. Gutierrez, Editor-in-Chief and Resident Data Scientist for insideAI News. In addition to being a tech journalist, Daniel also is a consultant in data scientist, author, educator and sits on a number of advisory boards for various start-up companies.
Sign up for the free insideAI News newsletter.
Speak Your Mind