In this talk, we will discuss self-supervised representation learning and then more specifically BYOL. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods had intrinsically relied on negative pairs, BYOL achieved a new state of the art without them. We will also describe follow-ups of BYOL that we have explored within DeepMind, BGRL (for graphs), MYOW (for new uncharted domains such as neural readings), and BAS/BEAST (for multi-modal domains). Finally, we will explore different takes on the analysis of BYOL and other open questions.
