Learning constraints from human stop-feedbacks in Reinforcement Learning

In standard reinforcement learning (RL) an agent optimizes its policy by exploring the environment via trial and error which can lead to potentially dangerous situations. In this work, we investigate an approach for enabling an RL agent to learn about dangerous states or constraints in the environment from stop-feedback, i.e., feedback preventing the agent from taking any further, potentially dangerous, actions. In many practical scenarios, we would assume that the stop-feedback is provided by human supervisors giving feedback on the RL agent's behavior while carrying out complex tasks in real-world scenarios, which, without this feedback, would require the design of sophisticated and overly complicated reward functions. To enable the RL agent to learn from the supervisor’s feedback, we develop a simple probabilistic model for approximating how the supervisor’s feedback is generated and consider a Bayesian approach for inferring dangerous states. We evaluate our approach in experiments using the OpenAI Safety Gym environment and demonstrate that our agent can effectively follow the imposed safety requirements. Furthermore, we gathered feedback from 100 human volunteers to validate our human-like feedback model and to obtain insights into the human provision of stop-feedback.