In this talk, we explore how Machine Learning (ML) and Algorithmic Information Theory (AIT), though emerging from different traditions, can mutually inform one another in the following directions:
AIT for Kernel Methods: We investigate how AIT concepts inspire the design of kernels that integrate principles such as Kolmogorov complexity and Normalized Compression Distance (NCD). We propose a novel clustering method based on the Minimum Description Length (MDL) principle, implemented via K-means and Kernel Mean Embedding (KME). Additionally, we employ the Loss Rank Principle (LoRP) to learn optimal kernel parameters for Kernel Density Estimation (KDE), extending AIT-inspired techniques to flexible, nonparametric models.
Kernel Methods for AIT: We also demonstrate how kernel methods can approximate AIT measures such as NCD and Algorithmic Mutual Information (AMI), offering new tools for compression-based analysis. In particular, we show that the Hilbert-Schmidt Independence Criterion (HSIC) can be interpreted as an approximation to AMI, providing a robust theoretical basis for clustering and dependence measurement.
Finally, we illustrate how techniques from ML and Dynamical Systems (DS)—including Sparse Kernel Flows, Poincaré Normal Forms, and PDE Simplification— can be reformulated through the lens of AIT.Our results suggest that kernel methods are not just flexible tools in ML— they can serve as conceptual bridges across AIT, ML, and DS, leading to more unified and interpretable approaches to unsupervised learning, the analysis of dynamical systems, and model discovery.This work is based on the following papers
Boumediene Hamzi, Marcus Hutter, Houman Owhadi, Bridging Algorithmic Information Theory and Machine Learning: Clustering, Density Estimation, Kolmogorov Complexity-Based Kernels, and Kernel Learning in Unsupervised Learning.
Boumediene Hamzi, Marcus Hutter, Houman Owhadi, Bridging Algorithmic Information Theory and Machine Learning: A New Approach to Kernel Learning.
Jonghyeon Lee, Boumediene Hamzi, Yannis Kevrekidis, Houman Owhadi, Gaussian Processes Simplify Differential Equations.
Lu Yang, Xiuwen Sun, Boumediene Hamzi, Houman Owhadi, Naiming Xie, Learning Dynamical Systems from Data: A Simple Cross-Validation Perspective, Part V: Sparse Kernel Flows for 132 Chaotic Dynamical Systems.