Dream to drive: Deep Learning for planning, prediction and control of self-driving vehicles

In this thesis I analyse Deep Learning techniques for model-based Reinforcement Learning with sensory data comprising of images and scalar variables. The methods are based on world models: nonlinear stochastic state-space models, implemented as artificial neural networks, that try to capture the spatial and temporal aspects of the agent/environment interaction. World models can be used to generate dream environments to learn a controller using Reinforcement Learning algorithms without the need of interacting with the real environments. Experiments are conducted in two OpenAI Gym environments and in the open-source CARLA simulator. The latter is used to investigate how world models might be applied in the context of self-driving vehicles, including approaches for action planning, for predicting various potential futures based on actions, and for choosing control inputs.