Researchers have developed a new analysis technique to extract significantly more information from galaxy surveys, delivering the most precise weak-lensing-only measurements yet of the Universe’s dark matter distribution.
Dark matter and dark energy account for around 95% of the Universe, yet their nature remains one of the greatest mysteries in modern science. Understanding them requires measuring how matter is distributed across the cosmos and how cosmic structures have evolved over billions of years.
One of the most powerful tools for doing this is weak gravitational lensing —the subtle distortion of distant galaxies caused by the gravity of matter lying between those galaxies and Earth. Modern sky surveys can measure these distortions for hundreds of millions of galaxies, but extracting precise cosmological information from such vast datasets remains a major statistical challenge.
Now, a team including researchers from the Department of Applied Mathematics and Theoretical Physics (DAMTP) in Cambridge, Imperial College London and University College London (UCL), have shown that combining physics-based analysis with machine learning can significantly improve what can be learned from these observations.
Reading the Universe through distorted galaxies
Weak gravitational lensing provides a unique way to map the invisible matter that fills the Universe and drives the growth of cosmic structure.
As light from distant galaxies travels towards Earth, its path is subtly bent by the gravitational pull of matter along the way. These tiny distortions alter the apparent shapes of galaxies by only a few percent, but when measured across tens or hundreds of millions of galaxies they reveal how matter is distributed throughout the cosmos.
The Dark Energy Survey (DES) measured the shapes of approximately 100 million galaxies across an area of sky roughly 25,000 times larger than the full Moon, creating one of the most detailed maps ever made of the large-scale structure of the Universe.
Traditionally, weak-lensing analyses compress these enormous datasets into so-called two-point statistics, which measure how galaxy distortions are correlated across the sky. These methods are robust and have underpinned modern cosmology for decades. However, they do not capture all of the information present in the data, particularly on smaller scales where the distribution of matter becomes highly complex.
Harnessing machine learning to learn more
Using data from the Dark Energy Survey, the team developed a new hybrid analysis framework designed to recover more of that information.
The new framework combines established cosmological statistics with machine-learning-derived summaries of the data, while rigorously accounting for uncertainties throughout the analysis.
"Using ideas from information theory, we can guide neural networks towards the most informative features in weak-lensing maps," says co-lead Lucas Mäkinen, Physics-AI Fellow in DAMTP, who originally developed the hybrid learning algorithm during his PhD at Imperial.
Starting from familiar observables such as the weak-lensing power spectrum, the researchers augment the analysis with machine-learning-derived summary statistics learned from realistic simulations of the Universe.
"I think of it as telling the networks where to look in the data," says Mäkinen, whose research is supported by the Cambridge-Infosys AI Centre (learn more about the Centre in our interview with Director James Fergusson).
Rather than replacing physical modelling, the machine-learning component is used in a controlled and interpretable way to identify additional patterns that are sensitive to the properties of dark matter and dark energy.
"This isn’t AI replacing physics," explains Imperial’s Professor Alan Heavens. "We use AI to compress information from around 100 million galaxy measurements into just seven highly informative numbers. Almost all of the relevant physics is captured in those seven quantities."
The work comes as cosmology enters an era of increasingly large and complex datasets. Next-generation surveys, including the European Space Agency’s Euclid mission and the Vera C. Rubin Observatory’s Legacy Survey of Space and Time, will observe billions of galaxies and generate datasets far larger and richer than those available today. Making full use of these observations will require methods that can extract as much information as possible while remaining physically reliable and scientifically interpretable.
"We found that combining physical insight with machine learning dramatically improves what AI systems can learn from cosmological data," says co-author Niall Jeffrey from UCL. "Rather than treating the algorithm as a black box, we can guide it towards the most informative features—and in doing so learn much more about dark energy."
Sharper measurements of dark matter and dark energy
Dark matter and dark energy dominate the Universe, but much of their imprint is encoded in subtle patterns that conventional analyses struggle to capture.
By recovering information that would otherwise be left behind, the new method can provide a sharper view of the dark Universe and a more powerful way to test competing theories of cosmic evolution.
One of the central challenges in cosmology is that different physical processes can produce similar observational signatures. Variations in dark matter density, matter clustering and dark energy can all affect the appearance of the Universe in related ways, making it difficult to disentangle their individual contributions.
The hybrid framework helps overcome this challenge by identifying subtle patterns in the data that allow these effects to be separated more effectively.
Applied to DES Year 3 weak gravitational lensing observations, the method extracts around three times more cosmological information than traditional approaches. Using the new framework, the researchers obtained significantly tighter constraints on key cosmological parameters, including the most precise weak-lensing-only measurements yet of both the matter density of the Universe and the strength of cosmic clustering.
Fully resolving a long-standing tension between weak-lensing observations and measurements of the early Universe, the results are also consistent with measurements from the Planck satellite, which mapped the Cosmic Microwave Background. Co-author Natalia Porqueres (CEA Saclay) notes: "In contrast to some earlier studies, results are in excellent and very precise agreement with measurements of the light left over from the Big Bang."
The findings have been submitted to Monthly Notices of the Royal Astronomical Society, and a preprint is available at ArXiv.
"We taught our neural networks the physics we already know, and then trained them to find the extra information hidden beyond it," Joshua Williamson, co-lead at UCL, says of the work. "We were amazed to see the result: nearly three times more information from the same data, and our most precise measurement yet of the matter and dark energy that shape our Universe."
As Euclid and the Rubin Observatory begin delivering unprecedented observations of billions of galaxies, techniques like this could help to answer some of the most fundamental questions in physics: what dark matter is, whether dark energy changes over time, and whether our current understanding of the Universe is complete.
This article is adapted from a news story by Imperial College London.
Top image: Detail from the Dark Energy Camera, for the Dark Energy Survey. The image combines the 1 millionth exposure with 127 earlier exposures to make this view of the field. Image credit: Dark Energy Survey/DOE/FNAL/DECam/CTIO/NOIRLab/NSF/AURA