Deep Reinforcement Learning and Control
Spring 2019, CMU 10403

• Liam Li: Tuesday 2pm-3pm, GHC 8133
  
• Shreyan Bakshi : Friday 3pm-5pm, GHC 5th floor commons
  
• Sarveshwaran Dhanasekar : Thursday 12pm-1pm, GHC 5th floor commons
  

• Class goals
• Schedule
• Resources
• Assignments and grading
• Prerequisites

Class goals

• Implement and experiment with existing algorithms for learning control policies guided by reinforcement, demonstrations and intrinsic curiosity.
• Evaluate the sample complexity, generalization and generality of these algorithms.
• Be able to understand research papers in the field of robotic learning.
• Try out some ideas/extensions on your own. Particular focus on incorporating sensory input from visual sensors.

Prerequisites

Schedule

Resources

Readings

1. [SB] Sutton & Barto, Reinforcement Learning: An Introduction, 2nd edition
2. [GBC] Goodfellow, Bengio & Courville, Deep Learning

1. Smith & Gasser, The Development of Embodied Cognition: Six Lessons from Babies
2. Silver et al., Mastering the Game of Go with Deep Neural Networks and Tree Search
3. Silver et al., Mastering the Game of Go without Human Knowledge
4. Mnih et al., Playing Atari with Deep Reinforcement Learning
5. Guo et al., Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning
6. Hasselt et al., Deep Reinforcement Learning with Double Q-learning
7. Mnih et al., Asynchronous Methods for Deep Reinforcement Learning
8. Houthooft et al., VIME: Variational Information Maximizing Exploration
9. Stadie et al., Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
10. Pathak et al., Curiosity-driven Exploration by Self-supervised Prediction
11. Burda et al., Large-Scale Study of Curiosity-Driven Learning
12. Kakade et al., A Natural Policy Gradient
13. Rajeswaran et al., Towards Generalization and Simplicity in Continuous Control
14. Lillicrap et al., Continuous control with deep reinforcement learning
15. Heess et al., Learning Continuous Control Policies by Stochastic Value Gradients
16. Ho et al., Generative Adversarial Imitation Learning
17. Andrychowicz et al., Hindsight Experience Replay
18. Nair et al., Visual Reinforcement Learning with Imagined Goals
19. Haarnoja et al., Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
20. Bagnell, An Invitation to Imitation
21. Nguyen, Imitation Learning with Recurrent Neural Networks
22. Bojarski et al., End to End Learning for Self-Driving Cars
23. Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
24. Ziebart et al., Navigate Like a Cabbie: Probabilistic Reasoning from Observed Context-Aware Behavior
25. Ho et al., Model-Free Imitation Learning with Policy Optimization
26. Finn et al., Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
27. Ziebart et al., Maximum Entropy Inverse Reinforcement Learning
28. Ziebart et al., Human Behavior Modeling with Maximum Entropy Inverse Optimal Control
29. Levine et al., Guided Policy Search
30. Levine et al., End-to-End Training of Deep Visuomotor Policies
31. Kumar et al., Learning Dexterous Manipulation Policies from Experience and Imitation
32. Mordatch et al., Combining model-based policy search with online model learning for control of physical humanoids
33. Rajeswaran et al., EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
34. Ganin et al., Domain-Adversarial Training of Neural Networks
35. Finn et al., Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
36. Rahmatizadeh et al., From Virtual Demonstration to Real-World Manipulation Using LSTM and MDN
37. Salimans et al., Evolution Strategies as a Scalable Alternative to Reinforcement Learning
38. Russo et al., A Tutorial on Thompson Sampling
39. Nikolaus Hansen The CMA Evolution Strategy: A Tutorial
40. Shaul et al. Prioritized Experience Replay
41. Hessel et al. Rainbow: Combining Improvements in Deep Reinforcement Learning
42. Pritzel et al. Neural Episodic Control
43. Jaderberg et al. Population Based Training of Neural Networks
44. Dosovitskiy et al. Learning to Act by Predicting the Future
45. Fragkiadaki et al. Learning Visual Predictive Models of Physics for Playing Billiards
46. Sanchez-Gonzalez et al. Graph Networks as Learnable Physics Engines for Inference and Control
47. Nagabandi et al. Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
48. Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
49. Rasmussen and Williams Gaussian Processes for Machine Learning
50. Deisenroth et al. PILCO: A Model-Based and Data-Efficient Approach to Policy Search
51. Oh et al. Action-Conditional Video Prediction using Deep Networks in Atari Games
52. Ebert et al. Self-Supervised Visual Planning with Temporal Skip Connections
53. Osband et al. Deep Exploration via Bootstrapped DQN
54. Blundell et al. Weight Uncertainty in Neural Networks
55. Kingma and Welling Auto-Encoding Variational Bayes
56. Doersch Tutorial on Variational Autoencoders
57. Kandasamy et al. Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
58. Brochu et al. A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
59. Goodfellow et al. Generative Adversarial Nets
60. Zhu et al. Reinforcement and Imitation Learning for Diverse Visuomotor Skills
61. Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
62. Tremblay et al. Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization
63. Muller et al. Driving Policy Transfer via Modularity and Abstraction
64. Chebotar et al. Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience
65. Peng et al. SFV: Reinforcement Learning of Physical Skills from Videos
66. Pathak et al. Zero-Shot Visual Imitation

General references

• Szepesvari, Algorithms for Reinforcement Learning
• Bertsekas, Dynamic Programming and Optimal Control, Vols I and II
• Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming
• Powell, Approximate Dynamic Programming

Online courses

• Rich Sutton's class: Reinforcement Learning for Artificial Intelligence, Fall 2016
• John Schulman's and Pieter Abeel's class: Deep Reinforcement Learning, Fall 2015
• Sergey Levine's, Chelsea Finn's and John Schulman's class: Deep Reinforcement Learning, Spring 2017
• Abdeslam Boularias's class: Robot Learning Seminar
• Pieter Abeel's class: Advanced Robotics, Fall 2015
• Emo Todorov's class: Intelligent control through learning and optimization, Spring 2015
• David Silver's class: Reinforcement learning

AWS Resources

• If you are not officially registered for this class, you are not allowed to request resources. We will be checking before we submit requests, so please do not request access to them.
• We will be offering AWS resources. All students should join AWS educate using this link: https://aws.amazon.com/education/awseducate/ using their @andrew.cmu.edu email address. If you do not use your andrew email address, your resources may be denied. You should do this as soon as possible, as it can take time to set up your accounts.
• AWS NOTE: You need to back this account with your own credit/debit card and we will give out allocation codes of $50, this is important as should you go over this $50 it will start charging to your card, please be sure to keep an eye on your funds and not forget to terminate instances. The university holds no responsibility in paying for additional usage.
• We will ask you to complete an Allocation Form in order to apply for your resources. This will be made available later in the semester. Note: HW1 does not require AWS resources.

Assignments and grading

Homeworks

Projects

Grace Day/Late Homework Policy

• Homeworks: Each student has a total of 4 grace days that may be applied to the homework assignments. No more than 3 grace days may be used on any single assignment. Any assignment submitted more than 3 days past the deadline will get zero credit. Grace days will be subtracted from both students in the homework team. E.g. an assignment submitted 1 day late will result in both team members losing 1 grace day from their total allotment.
• Projects: Each team will be allotted a total of 3 grace days on the project, separate from homework grace days (unused grace days from the homework assignments CANNOT be applied to the project). Project late days can be on the midway and final report, but not on the poster presentation. Any project submitted more than 3 days past the deadline will get zero credit.

Course Policies

Auditing

• Official auditing of the course (i.e. taking the course for an “Audit” grade) is not permitted this semester.
• Unofficial auditing of the course (i.e. watching the lectures online or attending them in person, but not turning in homeworks to grade) is welcome and permitted without prior approval. We give priority to students who are officially registered for the course, so informal auditors may only take a seat in the classroom is there is one available 10 minutes after the start of class. Unofficial auditors will not be given access to course materials such as homework assignments and exams.
• Please email Shreyan if you need further clarification.

Extensions

• Medical Emergencies: If you are sick and unable to complete an assignment or attend class, please go to University Health Services. For minor illnesses, we expect grace days or our late penalties to provide sufficient accommodation. For medical emergencies (e.g. prolonged hospitalization), students may request an extension afterwards and should include a note from University Health Services.
• Family/Personal Emergencies: If you have a family emergency (e.g. death in the family) or a personal emergency (e.g. mental health crisis), please contact your academic adviser or Counseling and Psychological Services (CaPS). In addition to offering support, they will reach out to the instructors for all your courses on your behalf to request an extension.
• University-Approved Absences: If you are attending an out-of-town university approved event (e.g. multi-day athletic/academic trip organized by the university), you may request an extension for the duration of the trip. You must provide confirmation of your attendance, usually from a faculty or staff organizer of the event.

Pass/Fail Policy

Online and Waitlisted Students

• All lecture videos will be recorded and made available online. We are currently working with the administration to try and create an online section for this course (10-703 B) so that we can work on getting everybody off of the waitlist and officially enrolled. Please be patient, as this may take time to complete. Past experience suggests that there will be sufficient seats in the classroom for everybody who wants to take the course, so we are optimistic that all students on the waitlist are likely to be able to register within the first few weeks.
• Waitlisted students should complete all homework assignments with the rest of the class.
• The first couple lectures are likely to be quite full -- so it'd be best for waitlist students to use the livestream. We'll let you know when seats start to become available. Once that happens, you are welcome to take a physical seat (GHC 4401) if there is an open one 5 minutes after class has started (e.g. 12:05pm)

Students with course conflicts

Academic Integrity (Read this carefully!)

Collaboration among Students

• The purpose of student collaboration is to facilitate learning, not to circumvent it. Studying the material in groups is strongly encouraged. It is also allowed to seek help from other students in understanding the material needed to solve a particular homework problem, provided no written notes (including code) are shared, or are taken at that time, and provided learning is facilitated, not circumvented. The actual solution must be done by each student alone.
• The presence or absence of any form of help or collaboration, whether given or received, must be explicitly stated and disclosed in full by all involved. Specifically, each assignment solution must include answering the following questions:
  1. Did you receive any help whatsoever from anyone in solving this assignment? Yes / No.
     • If you answered 'yes', give full details: ____________
     • (e.g. "Jane Doe explained to me what is asked in Question 3.4")
  2. Did you give any help whatsoever to anyone in solving this assignment? Yes / No.
     • If you answered 'yes', give full details: _____________
     • (e.g. "I pointed Joe Smith to section 2.3 since he didn't know how to proceed with Question 2")
  3. Did you find or come across code that implements any part of this assignment ? Yes / No. (See below policy on "found code")
     • If you answered 'yes', give full details: _____________
     • (book & page, URL & location within the page, etc.).
• If you gave help after turning in your own assignment and/or after answering the questions above, you must update your answers before the assignment's deadline, if necessary by emailing the course staff.
• Collaboration without full disclosure will be handled severely, in compliance with CMU's Policy on Cheating and Plagiarism.

Previously Used Assignments

Policy Regarding "Found Code"

Duty to Protect One's Work

Penalties for Violations of Course Policies

1. The penalty for the first violation is a one-and-a-half letter grade reduction. For example, if your final letter grade for the course was to be an A-, it would become a C+
2. The penalty for the second violation is failure in the course, and can even lead to dismissal from the university.

Accommodations for Students with Disabilities

Take care of yourself