publications

2022

1. NeurIPS Workshop

   Characterising the Robustness of Reinforcement Learning for Continuous Control using Disturbance Injection

   Catherine Glossop, Jacopo Panerati, Amrit Krishnan, Zhaocong Yuan, and Angela P. Schoellig

   In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022

   Abstract PDF Blog

   In this study, we leverage the deliberate and systematic fault-injection capabilities of an open-source benchmark suite to perform a series of experiments on state-of-the-art deep and robust reinforcement learning algorithms. We aim to benchmark robustness in the context of continuous action spaces – crucial for deployment in robot control. We find that robustness is more prominent for action disturbances than it is for disturbances to observations and dynamics. We also observe that state-of-the-art approaches that are not explicitly designed to improve robustness perform at a level comparable to that achieved by those that are. Our study and results are intended to provide insight into the current state of safe and robust reinforcement learning and a foundation for the advancement of the field, in particular, for deployment in robotic systems.

2. Thesis

   Benchmarking Reinforcement Learning for Safe Robotics: Constraints, Robustness and Transfer

   Zhaocong Yuan

   University of Toronto, 2022

   Abstract PDF Code

   Safe learning in robotics aims to deploy robots in real life with complex tasks and safety requirements. To push the agenda of safe learning, a crucial step is establishing common benchmarks that facilitate both reliable evaluations and research development. In this work, we contribute towards this goal by surveying the safe learning literature and proposing a versatile safe learning benchmark suite, safe-control-gym. The benchmark implements a variety of safe learning algorithms spanning control to reinforcement learning, it also implements critical features to support safety-relevant evaluations and algorithm development. With safe-control-gym, we conduct careful benchmarking on model-free reinforcement learning methods with respect to three metrics of safety: constraint satisfaction, robustness, and transfer performance. We envision safe-control-gym to provide a framework that brings various research together, and most importantly to accelerate the progress of safe learning in robotics.

3. IROS

   safe-Control-Gym: A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in Robotics

   Zhaocong Yuan, Adam W Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P Schoellig

   IEEE Robotics and Automation Letters, 2022

   Abstract PDF Code

   In recent years, both reinforcement learning and learning-based control—as well as the study of their safety, which is crucial for deployment in real-world robots—have gained significant traction. However, to adequately gauge the progress and applicability of new results, we need the tools to equitably compare the approaches proposed by the controls and reinforcement learning communities. Here, we propose a new open-source benchmark suite, called safe-control-gym, supporting both model-based and databased control techniques. We provide implementations for three dynamic systems—the cart-pole, the 1D, and 2D quadrotor— and two control tasks—stabilization and trajectory tracking. We propose to extend OpenAI’s Gym API—the de facto standard in reinforcement learning research—with (i) the ability to specify (and query) symbolic dynamics and (ii) constraints, and (iii) (repeatably) inject simulated disturbances in the control inputs, state measurements, and inertial properties. To demonstrate our proposal and in an attempt to bring research communities closer together, we show how to use safe-control-gym to quantitatively compare the control performance, data efficiency, and safety of multiple approaches from the fields of traditional control, learning-based control, and reinforcement learning.

4. Annual Reviews

   Safe learning in robotics: From learning-based control to safe reinforcement learning

   Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig

   Annual Review of Control, Robotics, and Autonomous Systems, 2022

   Abstract PDF Code

   The last half decade has seen a steep rise in the number of contributions on safe learning methods for real-world robotic deployments from both the control and reinforcement learning communities. This article provides a concise but holistic review of the recent advances made in using machine learning to achieve safe decision-making under uncertainties, with a focus on unifying the language and frameworks used in control theory and reinforcement learning research. It includes learning-based control approaches that safely improve performance by learning the uncertain dynamics, reinforcement learning approaches that encourage safety or robustness, and methods that can formally certify the safety of a learned control policy. As data- and learning-based robot control methods continue to gain traction, researchers must understand when and how to best leverage them in real-world scenarios where safety is imperative, such as when operating in close proximityto humans. We highlight some of the open challenges that will drive the field of robot learning in the coming years, and emphasize the need for realistic physics-based benchmarks to facilitate fair comparisons between control and reinforcement learning approaches.

2020

1. Thesis

   Emergent Communication Behaviors in Multi-Agent Systems

   Zhaocong Yuan

   University of Toronto, 2020

   Abstract PDF Code

   Multi-agent systems are prevalent in real life, powering the majority of people’s everyday activities. Multi-agent Reinforcement Learning is the study of these systems by means of machine learning, it has gained much attention in successfully applying to games, robotics and language understanding. This work concerns the emeregent behavior aspeect of MARL, speecifically emergent communications. We extend the MADDPG framework and experiment on a variety of multi-agent tasks from the eOpenAI Multi-agent Particle Environments, which span different behavior modes including cooperationo, competitiono, communication and mixture of those. Besides demonstrating the emergent behavior learnt from each task, we further provide analysis and insights regarding the MARL model design and training. At the end, we outline several possible lines for future work, hoping to inspire more multi-agent related research and advance the field of MARL.

2019

1. ICCV

   Meta-sim: Learning to generate synthetic datasets

   Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler

   In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019

   Awarded Abstract PDF Code Website

   Oral

   Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our work is to automatically synthesize labeled datasets that are relevant for a downstream task. We propose Meta-Sim, which learns a generative model of synthetic scenes, and obtain images as well as its corresponding ground-truth via a graphics engine. We parametrize our dataset generator with a neural network, which learns to modify attributes of scene graphs obtained from probabilistic scene grammars, so as to minimize the distribution gap between its rendered outputs and target data. If the real dataset comes with a small labeled validation set, we additionally aim to optimize a meta-objective, i.e. downstream task performance. Experiments show that the proposed method can greatly improve content generation quality over a human-engineered probabilistic scene grammar, both qualitatively and quantitatively as measured by performance on a downstream task.