Faculty researchers are working to equip artificial intelligence with human knowledge of physics to create powerful AI tools for a range of applications, from weather forecasting to engineering. Their efforts contribute to a body of work on AI that makes the Faculty a global leader in the field.
Artificial intelligence has come along immensely over the last few years. If you're in doubt try asking an AI to write your autobiography. Better still, ask it to write an autobiography of your cat.
A recent experiment along these lines revealed an impressive result: the book was brilliantly written with some passages uncannily, even spookily, accurate. That's even though the AI was only given a paragraph's worth of information about the cat in question.
But there were also the kind of glitches you'll be familiar with if you've played around with AI. One passage, exploring the cat's love for underfloor heating, read, "The tiles are cool against my paws, a welcome contrast to the warm floor." But that's nonsense. The whole point of underfloor heating is to warm up the floor tiles, and that's why cats love it.
AI without physics
The problem here was that the AI has no experience of the physical world. It learns to perform its task purely from data, in this case a huge volume of quality text written by humans. It reproduces the patterns it finds within language with amazing success. Sentences make sense and form a (mostly) coherent story. But the AI has no idea what its outputs mean. In particular, it has no knowledge of physics, and no way of checking whether its output makes physical sense.
When it comes to amateur writing such a lack of knowledge isn't earth-shattering. But there are also potential applications of AI that are all about the physical world. Examples are weather forecasting or applications in engineering, such as checking if an aircraft design is structurally stable. For these applications it seems desirable, even essential, to provide AI with some of the vast knowledge of physics we humans have accumulated over millennia.
This is why mathematicians, including researchers here at Cambridge, are currently busy building physics-informed neural networks. A neural network is a type of algorithm, originally inspired by the neurons in our brains. Neural networks are used in machine learning, the approach to artificial intelligence that has seen so many successes in recent years. The large language models that power ChatGPT, as well as models used for image generation and in areas such as weather forecasting, are all based on machine learning. Physics-informed neural networks are known as PINNs for short.
AI versus physics
Weather forecasting is a good example to illustrate PINNs. Traditional weather forecasts are based on our understanding of the physical processes that determine the weather. These are described in the language of mathematics, using differential equations (find out more in this article on numerical weather prediction). But because the weather is an extremely complex (actually a chaotic) system, and because the equations are incredibly difficult to solve, forecasts lack accuracy and require enormous computing power. This is why the Met Office has a very expensive supercomputer.
On the other hand we do have vast amounts of data about the weather. Machine learning algorithms could crunch through this data, learning important patterns during their training phase, such as how the amount of cloud in a region depends on parameters like temperature and pressure. Once the training is complete, they could then use this information about patterns to produce forecasts in the real world. That's the theory, but although there have been significant advances in using AI in weather forecasting, we're still some way away from replacing traditional methods (see this collection of explainer articles on plus.maths.org to find out more.)
AI with physics
The idea behind PINNs is to equip the algorithms with knowledge from physics while they are doing their learning. "If we have a differential equation that describes the [physical] system, we can incorporate that into the training phase of the neural network," says Georg Maierhofer of the Department of Applied Mathematics and Theoretical Physics (DAMTP). Maierhofer is also a member of the Maths4DL research project co-led by Carola-Bibiane Schönlieb, Professor of Applied Mathematics at DAMTP and Richard Nickl, Professor of Mathematical Statistics at the Department of Pure Mathematics and Mathematical Statistics (DPMMS). Maths4DL comprises researchers from the University of Cambridge, the University of Bath, and University College London and works on the mathematical foundations of machine learning.
There are various advantages to be had from introducing physics into machine learning. Traditional weather forecasting is limited by the knowledge we currently have about atmospheric processes — this is what the mathematical weather models are built on. These models are also always a simplification of reality. The weather depends on a multitude of factors whose interaction we don't always understand. Taking account of all of them in a mathematical model is an impossible task.
The advantage of a PINN is that it wouldn't have to rely on known physics alone, but could also mine the information hiding within existing weather data. It might therefore spot connections we don't know about, or which we don't know how to incorporate in the mathematical models. "The possible advantage of neural networks in this setting is that they might come up with some better, unusual approaches for these kinds of problems that might be more efficient, might be less computationally expensive, might be quicker, or more reliable in the long run," says Maierhofer. "This flexibility is one thing that is really nice."
In addition to finding new approaches, PINNs can also help with the physics that we do know. The differential equations that govern the behaviour of the Earth's atmosphere (the Navier-Stokes equations) are incredibly hard to solve. There's no chance of finding solutions with pen and paper or a few lines of computer code. Instead, a method called computational fluid dynamics is used in practice to provide approximate solutions, but this can require immense computing power.
The power of machine learning can be brought to bear here too, with PINNs providing alternatives to computational fluid dynamics for finding solutions to these tricky equations.
Hallucinations…
As it stands, PINNs are still in their infancy. They were first proposed in 2017 and research has only really taken off in the last five years or so. "There are significant opportunities, but [PINNs are not yet] able to compete in a large-scale industrial setting with anything that's been there classically," says Maierhofer. "We see some early advantages in using PINNs in a research-based setting but they are unable to scale up and provide the same reliability and certifiability that you have with classical methods."
One major challenge is that there aren't enough theoretical guarantees that a PINN won't give you false information. The large language models that drive ChatGPT (or the book-writing AI mentioned at the start of this article) can suffer from so-called hallucinations: they make up information that seems real, but is actually false.
"The same phenomenon can occur when you use machine learning methods to simulate physical systems," says Maierhofer. "For example when you want to simulate the flow of a fluid, then the machine learning method might give you something that to the eye looks correct, but might have the wrong pressure in certain areas. [In meteorology] this might mean it's missing a significant weather system over part of the UK, for example." In other applications, such as the design of aircraft, such a mistake could prove disastrous.
…and holes
Another challenge facing PINNs is that they can, quite literally, get stuck in a hole. During the training phase machine learning algorithms learn how to make their outputs more accurate. They do this by minimising a loss function, which measures the inaccuracy of their outputs. As a metaphor, imagine the algorithm feeling its way through a hilly landscape: at the peaks of the landscape the inaccuracy is high and in the valleys it's low. The aim is to reach the deepest valley of the landscape. The trouble is that the algorithm can only feel which way is up or down in its immediate vicinity. When it has reached the bottom of a small dip, it may then think it's reached the lowest point in the entire landscape, when really there's an even deeper valley further afield.
An illustration of gradient descent, the method which helps machine learning models to improve their outputs. The dot in the top right gets stuck in a small dip, rather than finding the deepest valley. Image: Jacopo Bertolotti
Once an algorithm has got stuck in this way, no amount of computing power will help it out. That's in stark contrast to classical methods. "In classical methods there's this paradigm that if you increase the computational resources, by letting the algorithm run [longer] or giving it a few more computers to run on, you typically get a more accurate answer," says Maierhofer. "In machine learning methods there's a barrier that currently people cannot break. It seems that after you've reached a [certain] point of giving the computer more resources, these extra resources [no longer help]. With current methodologies this is a big problem."
A global leader in AI
Solving these problems means working on the theoretical underpinnings of machine learning — a welcome challenge for mathematicians that has been taken up by many other members of DAMTP and DPMMS. Their work finds applications in the areas that Maierhofer mentions, but also in medicine and public health, ecology, astronomy, and biology. Faculty members' research also contributes to Polymathic AI, an initiative that seeks to develop an AI tool for general scientific discovery. And because AI poses a societal challenge, the Faculty is helping to shape policy decisions, for example through John Aston's role as Professor for Statistics in Public Life.
This combined effort puts the Faculty in a world-leading position when it comes to developing the power of AI. If we are going to make sure that AI tools are deployed, not just effectively, but also safely and responsibly, then the involvement of those who can build them from the ground up is essential.
A version of this article first appeared on plus.maths.org, a free online platform which is part of the MMP, the Faculty's award-winning outreach programme.
You might also want to listen to this episode of the plus.maths.org podcast which features Maierhofer talking about PINNs.
The image of the kitten above was generated by AI using Adobe Firefly.