We live in an age of increasing complexity, where accelerating technology and global interconnection hold more promise – and more peril – than any other time in human history. As well as financial crises, issues around climate change, automation, growing inequality and polarization are all rooted in the economy, yet standard economic predictions fail us.
In this interview, I speak to J. Doyne Farmer, who is an American complex systems scientist and entrepreneur who was a pioneer in many of the fields that define the scientific agenda of our times: dynamical systems, chaos, complex systems, artificial life, wearable computing, time series analysis, theoretical biology, and the theory of prediction. Currently he is Director of the Complexity Economics programme at the Institute for New Economic Thinking at the Oxford Martin School, Baillie Gifford Professor of Complex Systems Science in the school of Geography and the Environment at the University of Oxford, Senior Associate Research Fellow at Christ Church College, Chief Scientist at Macrocosm, and an External Professor at the Santa Fe Institute.
Previously, he was an Oppenheimer Fellow and the founder of the Complex Systems Group at Los Alamos National Laboratory. While a graduate student he led a cooperative calling itself Eudaemonic Enterprises who built the first wearable (and concealed) digital computer and used it in casinos, successfully beating the house. He was a founder of Prediction Company, an early quantitative automated trading firm that was sold to the United Bank of Switzerland in 2006. His current research is in economics, including agent-based modelling, financial instability and technological progress, and a founder of Macrocosm, a new company using complexity economics to guide the green energy transition.
Q: What is a complex system?
[Doyne Farmer]: A complex system is one where the properties of the building blocks, when they interact, can create phenomena that are very different from the building blocks themselves. The human brain is a great example. You take a neuron, which is a cell – you put an electrochemical signal in, electrochemical signal comes out – but somehow you hook 80 billion of them together and you get a brain. You have to hook them together just right, of course, but the result is completely fundamentally different than what you started with. So fundamentally different that it wasn’t until the 20th Century that people could believe it was even possible.
The economy is another example, noticed even earlier by Adam Smith. Rather than all being Robinson Crusoe (which would leave us with a very subsistence, primitive way of life), by specializing and interacting through the social mechanisms we create, we can do amazing things. It’s that huge amplification of effort that the economy is capable of that makes it a complex system.
[Vikas: is that why our standard model of economics doesn’t seem robust enough to predict major market events?]
[Doyne Farmer]: The standard models were formulated through a process that started well before computers were in place, and I would say it’s undergone a certain lock-in. Once you start going down that path, it’s hard to break out of it to another path.
As a result, economics is stuck. It’s not even that the existing models are wrong, they’re just very limited in what they can do, and mainstream economists have gotten very locked in to using those models – and only those models. So somebody like me who advocates simulation is a heretic within the academic economics community.
Q: Why are people so unwilling to adopt complexity models?
[Doyne Farmer]: Well, people get invested in what they’re doing. There’s a vested interest – if the ideas you’re teaching and have worked on are completely abandoned in favour of something else, then your legacy goes to zero. I think that’s a big force. And maybe more people are trained to think in a certain way, they’re trained to believe that’s the right way to do it. They really do think their way of doing it is the best way of doing things.
That said, I think the biggest problem is with academic economists. A lot of central banks are fairly open, because central banks are on the front line – they get stung if things don’t work. They’re more interested in plurality views. So some of the central banks are starting to break out. Bank of Canada is using a complexity economics macro model side by side with their other models. Bank of Italy is developing one, Bank of Hungary is developing one, the Bank of England have been seriously interested in things like this.
So some of the central banks are really branching out. And many of the central banks now, I would guess at least 8 of them, have agent-based housing market models that build on models we built earlier. So it’s beginning to happen.
Q: How do you set parameters (and boundaries) on a complexity model?
[Doyne Farmer]: …you have to keep the model simple as long as it’s not too simple. So you want to get all the essential things in there but you don’t want to have it cluttered up by other stuff. And since it’s an economic model, we have to model human decision making, but we only need to focus on decision making that relates to economics. How much will people spend, how much will farms produce, what prices will they set? Those are the questions we need to answer, and we need to focus on the behaviour that affects those things.
So that already eliminates 99.9% of what we’re actually doing with our brains, and in a way the economic part constrains everything too. There’s a lot of conservation rules in economics, accounting is a constrained space. And so that helps a lot.
Agent-based modelling helps us go beyond just mathematics. We may have some equations inside of our agent-based models, but we can do things that are not easy to write down with equations. Because really with a computer program, any process you can specify, we can write down and put in there. And computers deal with computation much better than we humans do.
So part of the reason I’m arguing the time is right is because unlike back in the 60s when these ideas about complexity economics were first floated by people like Herbert Simon, we now have all the tools to do it. Computers are a billion times more powerful, the data is vastly better, our understanding of psychology is vastly better, we know a lot more about how to program models like this – we have all the elements we need now to do these things and to do them well.
Q: How do we make the connection between say- biological complexity- and economics?
[Doyne Farmer]: So, my observation is if we make a parallel between biology and economics, the role of the economy is like the role of metabolism in an organism. Because what does the metabolism do? It takes in food, resources from the environment, and breaks them down and rearranges them to make other things that we need – like ATP for energy and proteins as building blocks and so forth.
Similarly, the economy takes in natural resources from the environment, combines them with labour to make goods and services that we use, and go through a cycle that we then eventually go away and put in the junkyard or to landfill. And so there’s the same kind of cycle of processing natural resources to turn them into things that satisfy human needs that we see in a metabolism.
Q: What are the distortions that credit creates in the market?
[Doyne Farmer]: Well credit, as you say, makes the world go around, but we want the right amount of credit. Too little credit and the economy can’t grow, too much credit and the economy becomes unstable and we have the great financial crisis. So we need to find that proper Goldilocks point in the middle. And the Goldilocks point may also move depending on other conditions in the economy.
I think complexity economic models can help us better understand that, because we can understand the systemic effects that credit can create, which is what causes the problems. So for example, if a hedge fund borrows money to buy assets – how can they use borrowed money to buy assets? Well we do that all the time. We borrow money to buy a house, people borrow money to create a business, people borrow money to buy assets.
But if there’s a price drop in those assets, the lender will tell them okay, you need to pay some of the money back. How do they pay the money back? They sell the asset. And if they were the only ones doing that, that would be fine. But if everybody’s doing that, everybody sells assets, prices go down, which causes more asset selling, which causes prices to go down, which causes more asset selling. So you can get an enormous amplification from small little bits of noise.
And we show in similar models I talk about in the book – very simple little models – how that works and why it’s dangerous and how we need to properly model that system so that we know well this is about the right amount of credit.
Q: Can complexity help us understand how markets respond to news events?
[Doyne Farmer]: Yeah, I mean we see under traditional conventional markets theory, markets should only react to new information. And they should actually correctly process that information, understand what it means, and how it should affect prices. But in reality, errors are made. Effects like the one I mentioned happen whereby there are mechanical amplification of noise that can destabilize markets.
And we see that when a market gets volatile it tends to stay volatile for a long time, because it’s feeding back on itself. Price volatility creates uncertainty which creates price volatility. And quiet markets create calm which propagates quiet markets. So the models we’ve built explain why that happens.
It can happen because you have trend followers who say if prices are going up they’re probably going to keep going up so I’m going to bet, and they bet by buying the asset which pushes prices up – again a self-reinforcing feedback loop. Or leveraged investors who respond to a downward glitch in prices by selling, which drives prices down which causes even more buying and crashes.
So we have some understanding of what the mechanisms are that are creating these effects and which I frankly don’t think the mainstream has good explanations for these things.
Q: Can complexity help us better regulate?
[Doyne Farmer]: It’s challenging, because to really do that right you have to gather a lot of data, you have to use a fairly big project, you have to be careful that you don’t end up with things like insider trading as a result of that, so there’s a lot of nervousness about that. Nervousness about confidentiality.
But yes, I believe that the governments have a unique role because they have access to all the information, or at least the information about people’s balance sheets. They don’t know what they’re thinking, hopefully, but they can see what they’re buying and selling and they can see the whole picture.
And so being able to see like that can be very helpful. It can also be very helpful for testing things out – like suppose we had that and said here’s this new thing called a mortgage-backed security, what happens if everybody buys them? I think we would have seen quite easily that we would have gotten the result we got, and we could have then done it in moderation.
Q: How can complexity help us better model climate change?
[Doyne Farmer]: I think this is ideally suited for that, because climate change is another example of something that’s knocking the system out of equilibrium. We’re undergoing major transformation in our energy system. It’s just beginning but it is beginning now, and I think it’s going to happen quickly wherein we have to plan carefully or we could have a very bumpy ride.
It’s going to cause geopolitical rearrangements and so I think we need to really use all the tools we can to try and get some guidance about where to go, and in complexity economics models we can put in the type of details we need to have – like the difference between different types of energy technologies, the geographical components and so on – to get more faithful models of how things are likely to unfold.
The climate problem is fundamentally an economic one. It stems from our methods of energy production and food production. To stop worsening climate change, we need to transform these systems.
Modelling the economy is challenging, partly because we lack the kind of fundamental laws that we have in physics. However, in some ways, it’s actually less complex than modelling weather. Weather systems are governed by highly non-linear equations that create instability – which is why you can have hurricanes and doldrums occurring in neighbouring regions.
The economy, by contrast, has about 200 million firms worldwide. While that’s a large number, it’s well within the processing capabilities of modern cloud computing. This means modelling the economy at a 1:1 scale is actually feasible.
Q: How will AI impact the practical application of complexity?
[Doyne Farmer]: Let me clarify the difference between AI models, machine learning models like you’re talking about, and agent-based modelling.
In machine learning, you take a bunch of data and apply some non-linear functions to the data, such as an artificial neural network, and then use that to make predictions. In contrast, in an agent-based model, you build a model from the bottom up using your knowledge about how the system works.
If you want to examine a situation where the world is likely to change significantly, AI models may struggle because they are built on the world as it is, and it’s unclear if they can handle an alternative situation where they haven’t seen data. On the other hand, you have much more hope with an agent-based model, which has the causal mechanisms built in. Agent-based models can also work with less data.
At the same time, the two approaches are friendly cousins because, as you’re pointing out, you can use AI in many useful ways. For example, if you want to look at which companies use which patents, you can use a large language model to read descriptions of companies and patents and match them up. Several people these days are working on using large language models to model the way real people think. In other words, not what is the best solution, but what a person would decide to do in a given situation, or even a person of a certain kind. This is another great area where AI and agent-based modelling can work together.