Cybernetics: The Science Economics Should Have Become – with Dan Davies

Danny Buerkli: My guest today is Dan Davies — an economist, contributor to the FT, former investment bank analyst, and author of several books, including The Unaccountability Machine: Why Big Systems Make Terrible Decisions and Lying for Money: How Legendary Frauds Reveal the Workings of the World. Dan, welcome.

Dan Davies: Hi — thanks very much for having me.

Danny: What is the optimal level of fraud in a society?

Dan: Well, it’s hard to know what the actual optimal level might be, but the optimal level is certainly not zero. Because the only way that you can get down to zero fraud is by restricting economic activity to only those things that you can absolutely, certainly check out. And that’s something we do actually see in a few low-trust societies. Famously, if you go and try to set up business in Lebanon, you need contacts there, because you need people who can check you out — check your family out to, the local joke is, three years before the birth of Christ. And if you can’t do that, no one will do business with you. And that’s one of the reasons why the economy of Lebanon ends up in a much less prosperous position than it could otherwise be.

On the other hand, you can actually see that the equilibrium level of fraud — which is slightly different from the optimal level — can be very high and still have pretty good outcomes. So you get somewhere like Canada. Canada is absolutely famous for having a surprisingly high level of financial fraud and financial crime for the size of the country. There’s an example that came out after I published my book — which is a shame, because it would have made such a perfect example. McMaster University, I think, in Western Canada, was building a new campus. And one day they got an email that said: hi, here are our new bank details.

And they had transferred ten million Canadian dollars into that bank account before the actual contractor phoned up to ask why they hadn’t been paid. And that sort of thing does go on in Canada. If you show up with a nice suit and you’re polite and you’ve got a nice smiley manner to you, then you will find loads of people who are prepared to do business with you, who will extend you credit — and you can change bank account details with a single email rather than with five weeks of process. That does make them vulnerable to fraud. But then you have to ask yourself: would I rather have the economy of Canada, or would I rather have the economy of Lebanon?

I mean, the question here — since we are economists of one kind or another — is: do you want to protect yourself against fraud, or do you want to be rich?

Danny: Right — presumably, I imagine, you’d go clearly for the latter. Essentially anyone would, hopefully. Which is your favorite financial fraud? Because in the book you do ultimately say that there’s actually nothing admirable about this — the longer you stare at these frauds, the less impressive they are. But surely there must be one where you’ve just got to acknowledge and admire the sheer craft of it.

Dan: It’s not so much the sheer craft of it. But the one that really sticks in my mind — and the one I have a sneaking affection for the story of — is the great salad oil swindle. It was a guy who ran a massive salad oil trading business, and he financed that trading business by borrowing money collateralized against the high-quality soybean oil which he held in tanks somewhere else in New Jersey. And he, for various reasons, was a crook, and he wanted to borrow a lot more money than the soybean oil in those tanks would have supported. And so he needed to attempt to own a lot more oil than he had.

But oil floats on water. And so when someone is going out to inspect your tanks, it’s quite hard for them to tell the difference between a tank full of extremely valuable salad-grade vegetable oil and a tank that is basically full of seawater with a few gallons of soybean oil floating on top. And this guy — Tino De Angelis, his name was — ran that scam for a surprising amount of time. And it got quite sophisticated, because he also had a system of pipes to send oil from one tank to another, so that the inspector could see the same salad oil showing up in one tank. I wouldn’t say it’s admirable, but it’s got that kind of slapstick comedy quality to it that just stuck in my mind.

Danny: That’s pretty good. Not quite fraud territory, maybe — but if we look at the financial system, which you’re deeply familiar with: why did Credit Suisse go under?

Dan: Well, I’ll put my cards on the table here. I used to work at Credit Suisse — I used to work at Credit Suisse when that was a thing that you could say with some sort of pride — in the London office between, I think, 2007 and, I want to say, 2012, but it might have been 2011. And the thing about Credit Suisse is that we were cowboys. We were cowboys and prop traders, in what I would say is the most positive sense that you can have of us: it was very much a prop trading culture.

Everyone knew that the bank ran on proprietary trading, and everyone had a culture and a system of values that was based on how you make money and how you manage your risk as a proprietary trader. And so it was very much a firm in which loss-making positions were cut immediately and ruthlessly. It was a firm which paid huge amounts of attention to how well its business was diversified. And this basically comes down to the fact that we knew that we were cowboys, and we knew that we had to be managed like a bunch of cowboys. Then the global financial crisis comes along.

Now, that was about the time I got out of Credit Suisse. Credit Suisse had to change its business model. Proprietary trading was out, and now you had to do a load of client business. And the thing about that is that client business requires a very different risk management philosophy from proprietary trading, because it’s not arm’s-length, it’s not transparent and anonymous — it’s about relationships and trust. So Credit Suisse went heavily into fee-based business, like asset management and prime brokerage.

And these were considered to be low-risk areas of business, because you’re not taking proprietary risk on the price of the securities that you trade. Unfortunately, the thing about those low-risk businesses is that the only way that you can lose money in them is if you’re dealing with a crook. And Credit Suisse were cowboys. We were not relationship managers. We were not set up to detect crooks.

And so Credit Suisse got into the Greensill funds, and it was prime broker to Bill Hwang. And all of these things would never have shown up in a proprietary risk-taking culture, because you would never have got into a position that you can’t get out of. In a relationship-management, fee-income-based culture, you have to get into those sorts of relationships, and you have to understand that if the deal turns bad, you’re stuck with it. You can’t trade out of it.

So, to my mind, what killed Credit Suisse was the fact that it had to — for post-crisis regulatory reasons — change its business model to something that it was just culturally not equipped to deal with. And this shows us another paradox of the industry, which is that often low-risk business can actually be more dangerous than high-risk business. Because risky stuff like proprietary trading — everyone knows it’s risky and manages it that way. Low-risk stuff like prime brokerage and asset management — everyone presumes it’s low risk and doesn’t manage it as aggressively. And that lets it grow to the point where it can kill the bank. I’ll keep on telling everyone: we had a mortgage bond crisis in AAA securities. We didn’t have a high-risk risk-taking crisis because people were trading tech-stock equities.

Danny: Right. If we turn to your other book, The Unaccountability Machine, which is about the ideas of cybernetics: I want to get into some of the details in a minute, but you present cybernetics essentially as a competitor idea — as a competing framework — to how we think about the economy. And I’d love for you to explain why you think that is the case.

Dan: Yeah, absolutely. Cybernetics — specifically management cybernetics, which is the application of the theory to organizations — is, to my mind, the kind of science that economics should have turned into in the industrial age after the Second World War. Because cybernetics is basically the application of the mathematics of information theory to questions of economic organization and management. The trouble that economics had is that most of the important writings about the economics of information — things like Hayek and the socialist calculation problem, things like Ronald Coase and the theory of the firm — these were all done during the 1930s. And that means that most of the bedrock of economics and information is based on work that was carried out before Alan Turing.

It was before Claude Shannon. It was before there really was a mathematically rigorous concept of information to base it on. And as a result, all through the time that people started trying to apply rigorous Shannon information theory to economics, the profession of economics thought that these were settled issues. They thought all this stuff was settled in the 1930s. And so economics never really dealt with some of the challenges of information theory.

Economics felt like it had solved all the problems it had relating to information — with Friedrich Hayek and the socialist calculation problem demonstrating that the market could solve problems which a central planner couldn’t, and Ronald Coase on the theory of the firm explaining that the boundaries of the firm were determined by the transaction costs of information. But they were formulating all these questions before Claude Shannon, before Kurt Gödel and Alan Turing. So they were actually solving these problems before there was any really rigorous concept of information. And that came back to bite them in later years, as the world got more complicated. Because once you have something like Shannon’s information theory, you start to realize that the biggest problem an organization has is in supporting enough bandwidth to transmit and process all of the information that it’s receiving from the environment.

To put it in less abstract and theoretical terms: every day you sit down at your desk and it’s like drinking from a fire hose. You’re getting so much information, and you’ve got a limited ability to process it yourself. And so every day you are deciding how you are going to deal with that — what you’re going to pay attention to. And every decision about what you’re going to pay attention to is also a decision about what you’re going to ignore. And so you have to come up with a representation of the world — a sort of mental model — and then you make decisions about your imaginary mental object, and you hope that they are going to match up to the real world.

And then you get feedback from the real world about how well your mental model works. And that’s something that’s surprisingly difficult to formulate in the language of economics, because economic models are almost always working with perfect information, perfect rationality, and sometimes with perfectly private information. There isn’t much room, because of the way that the theory develops, for this world in which you’ve got huge amounts of public information but you’re only able to process a subset of it. And as the world and as organizations have got more and more complicated, that intermediate case — which economics doesn’t deal with too well — has become more and more important. And a picture of the world which leaves it out has become more and more inaccurate.

Which is why I started writing the book and trying to make the case that we should go back to some of these rather obscure engineers and management scientists from the fifties and sixties, and start really trying to say: how would our theory of organizations and our theory of economic activity change if we started taking partial information and uncertainty seriously, and absolutely putting it at the heart of everything that we do?

Danny: Now, one of the laws, quote unquote, that you use — that’s part of the cybernetic toolkit — is Chandler’s law, right? Complexity grows exponentially; management capacity doesn’t. Why wouldn’t LLMs change that? Don’t LLMs give us access to essentially infinite excess analytical capacity?

Dan: Interesting question — and I have changed my mind about this a couple of times over the last year. One thing I should say for the benefit of your readers is that I think I did write exponentially in the book, and got absolutely roasted at a seminar for doing so. In fact, complexity grows quadratically: it grows with the number of things that you add. To my mind, that doesn’t matter.

It’s still a growing curve, which is always going to overtake a straight line. But technically, yeah, fair dos, I will go hands up: I wrote exponentially; I should have said quadratically. We’re talking about Alfred Chandler’s theory of organizational growth here, from his great book Strategy and Structure, which says that organizations grow, they get more complicated as they grow, and the capacity to manage that complexity grows at a slower rate than the complexity, because it’s linear growth compared to quadratic. And over time, this means the complexity of the organization will outstrip the ability to manage it — at which point some kind of reorganization is necessary, or crisis is inevitable.

It’s definitely the case that technological revolutions can change that trade-off for a while. Chandler himself writes that the telegraph and the railway marched across America in step, because we needed the electronic telegraph to be able to handle the complexity of an organization as big as the transcontinental American railroads — which were probably the most complicated organization there was at that time. And you had similar technological revolutions with the electronic computer. Even the filing cabinet was an information technology revolution in its time. Even the humble filing cabinet was a big deal in terms of making it possible to manage more complicated organizations.

So was double-entry bookkeeping. The thing is, though, that these technological revolutions historically have always meant that you’ve got a step change in that linear function. So it buys you a few years. So with LLMs, we might certainly be able to buy ourselves a few years, or even buy ourselves a few decades — which is absolutely not nothing. It’s really important.

In the end, though, you use that technological capability to grow, and you use it to get more complicated. One of the things that I’ve been writing in the current book that I’m working on is that when you reach bottlenecks, that’s not a natural phenomenon. It’s not an obstacle presented to you by an uncaring universe. A bottleneck is the consequence of expanding until you reach a bottleneck. And that’s where I think LLMs basically lead us.

The interesting question to me — and the one that I have changed my mind on a couple of times and been working on for the last year and a bit — is whether we carry on with the model that we seem to have at the moment, which is to just drop the LLMs into our existing organizational structures and hope to use them to replace human beings, on the idea that you can spin up the equivalent of a hundred employees on a server rack for the cost of one employee. Or whether we actually take this challenge seriously and reorganize the entire organization around the new technology. Because what someone like Alfred Chandler would say is that reorganization of some sort is inevitable. And the really, really interesting question there is that any such reorganization probably means changing things so that the machine is actually taking the decision. You’re going to have to actually have decisions where the human in the loop doesn’t exist.

Because if the human in the loop is still there, the bottleneck will still be there, and there’s a limit to how much can be done. But if you want to take the human out of the loop and really start gaining the benefits of using LLMs to run extremely large and extremely complex organizations, then you’re taking a big risk. And you’re taking a step that lots of people are not comfortable with — and lots of people are not comfortable with for really, really good reasons. Because we don’t have a legal framework to say who is accountable for a decision that no human being made.

Danny: Is that true, though?

Dan: At the moment, the only legal framework we have is that the last human being to touch it owns the consequences.

Danny: And maybe more abstractly: as long as we have legal entities that are still ultimately owned by humans, then there’s still a human accountable somewhere. I think it would be different if we had legal entities that were not owned by humans. That would probably be very bad indeed.

Dan: This is the challenge. And this is why I don’t think we’re necessarily going to be able to have our solution of organizing everything purely around the AIs. Because as long as we have that human who is accountable, that human is also the bottleneck — unless you can find some class of human beings who are prepared to take legal responsibility for decisions that they have no control over.

Danny: Right — as a sort of well-compensated human punching bag.

Dan: Exactly. And maybe you will do that — because, to an extent, lots of CEOs today exist to take responsibility for decisions made in the organization that they have much less control over than they think they do. So I think there’s a really interesting set of questions there that go beyond technology and science into some really fundamental questions about politics, legitimacy, and ethics. There are some people I’ve been talking to over the last year who are doing incredibly interesting work in this field.

Danny: Interesting. Can we just push on this for a minute? Because I’m not sure. As you say, already now CEOs — people who run large organizations — are accountable for things that they cannot possibly know are going on, and we occasionally make them step down for decisions that they had absolutely no way of knowing about. And it’s not obvious to me that that is fundamentally different from what you’re describing, which is an LLM taking a decision that no one has had a hand in.

Dan: It is kind of similar. I started off making the case in The Unaccountability Machine that organizations are artificial intelligences — they are non-human decision-making systems. And I think you’re right that that is the system we have at the moment. But here’s the question: is that system working? Are we really saying that everyone’s really happy with that system?

To take an example I’m most familiar with, because it was a formative period in my life: look at the collapse of Northern Rock, or the collapse of Royal Bank of Scotland. Look at the LIBOR scandal, even more so. The decisions taken which created the LIBOR scandal were definitely decisions that were not made by the chief executive of Barclays or the chief executive of RBS. And they did have to stand down from their jobs for doing so. Was anyone happy with that?

The hell they were. Everyone immediately said: well, all he’s doing is standing down from his job. He should lose his pension. He should be thrown in jail. And you can’t do that in a democratic country, because if you want to put someone in jail, you have to actually prove that they broke the law and committed a crime.

But the degree of accountability we actually have found it possible to have there is one that a lot of people are really unsatisfied with. And this is something I do try to make the case for in The Unaccountability Machine: that popular dissatisfaction with the amount of accountability that it’s possible to apply, even to the organizations we have today, is, I think, at the roots of a lot of the problems of populist politics, and at the roots of a lot of popular dissatisfaction with the system as it is. So although I agree with you that, from a technocratic, technological point of view, we could probably go further with the system that we have now — pushing more decision-making to machines and having the human beings there taking some kind of semi-ownership of the decisions for legal purposes — I don’t necessarily think that’s politically possible. I think we’ve got a huge problem of tolerance and consent with the system as it is.

And so if we push it even further in that direction, we are going to get a backlash even worse than what we already have — one that we might not be able to deal with.

Danny: So it sounds like we’re between a rock and a hard place. Because on the one hand, there’s the call for more accountability — and possibly no good idea for how to create it. On the other hand, what you’re saying is: if we want to reap the benefits, you probably do have to devolve decision-making powers to the system. Have you come across any good ideas for how to get out of this conundrum?

Dan: I think, personally, that the idea is the old one. It’s the one that comes from Alfred Chandler; it’s the one that comes from guys like Stafford Beer and all of those wonderful sixties management theorists — which is that the only solution to complexity is decentralization. The only solution, for a society that is going where we are, is to push the decision-making further down the scale, and leave the top of the hierarchy making much fewer operational decisions that directly affect people’s lives, and much more abstract decisions about general policy and general direction. And so you actually have to start allocating the resources to build consent.

You have to start allocating the resources to make it possible for people to be involved in local decision-making — or to almost have a formal signing ceremony where people who don’t want to be involved in decision-making take accountability for that, and promise: I have decided I don’t want to be involved in this decision, so I’m not going to complain when the results go a different way. And so we end up towards where all of the people talking about participatory democracy, citizens’ assemblies, and widespread consultation are. It is really noticeable to me that the places in Europe and in the world which have done the best in terms of maintaining public consent and avoiding populist politics have been the smaller states of Europe, and the states where there’s much more history of decentralized government and community involvement. Places like the Nordic countries, Ireland, Luxembourg, Switzerland. None of them have been completely immune, because it’s a massive global trend, but they’ve kept the general consent in the system at a much higher level than we’re seeing in bigger and more centralized places like the United States and the UK.

Danny: And there’s an obvious cybernetic interpretation — Ashby’s law of requisite variety, which we should get to in a second. I think one way of understanding what local democracy does is that it satisfies Ashby’s law of requisite variety, right? It doesn’t reduce or dampen variability, but it allows for it — and so you get fewer of the problems that you see elsewhere.

Dan: Yeah — it just kind of makes things manageable. Ashby’s law, for people playing along at home, is that anything which wants to regulate a system has to be able to represent the complexity of that system. It’s kind of like Shannon’s theorems about bandwidth. It’s like the idea of getting a drink from a fire hose, from a management science point of view. It’s just this idea that you can’t control something at any finer granularity than the mental model you have.

So when you have a centralized government: if I go to the Prime Minister of the UK and say, should we build the new bus stop in Tiverton or in Crediton — those are two towns in Devon — I would not expect the Prime Minister of the United Kingdom to know the difference between Tiverton and Crediton, because they are small towns, and he doesn’t operate at that level of resolution. If you go to Devon County Council, everyone knows where Tiverton is, everyone knows where Crediton is, and everyone has a mental model in which they can discuss the question of locating a bus stop in one of those two places. Problems happen when you over-centralize governance of any kind, because then you are asking the central function to answer questions that they don’t really have the capacity to represent.

And so then they’re always scrambling, trying to come up with bits and pieces of information. This is the failure mode of over-centralized systems, which we all know as micromanagement. And things have to be pushed down to the lowest level where the decision can be taken, because that’s the decision-maker which has a mental model of the world that includes the necessary level of detail. This is the sort of question that really should have been part of the science of economics for the last forty or fifty years. The appropriate level of delegation and the appropriate level for strategic planning is just one of the fundamental questions of organizations and management.

And it’s been left to the business schools, and it’s been left out of economics. And I figure it really needs to be brought back in.

Danny: You have a great line in the book about the practical application of Ashby’s law of requisite variety, and how to use it to spot a doomed project. How do I know if a project is doomed or not?

Dan: You just literally draw up in your head an information balance sheet. I adapted that from a great line of Stafford Beer’s, where he says that the information balance sheet will always balance — it’s just that you need to hope that it balances in a nice way rather than a nasty way. If something is being set up which doesn’t respect the principle that the regulatory system has to be able to represent the complexity of what it’s managing, then something will go bad. You will first see the signs of this going bad because that management system will start behaving like a system under stress — because they are under stress, because they are being faced with more detail than they have the capacity to represent.

When people or systems are in that position of overload, they do exactly what a telecom switch does when it’s overloaded, which is that they start dropping bits of information. I had a great old boss who used this line — which I’ve taken and repeated in a variety of contexts — where he said that stupidity is an affliction, but ignorance is a strategy. And ignorance is the information processing system of last resort. It’s what central management functions do when they are overloaded and they are unable to represent the complexity of what they’re managing: they start throwing bits of information away.

And at that point, pretty much with probability one, in finite time they’re going to throw away a bit of information that was important, and they’re going to do a dumb decision. And furthermore, it is going to be surprisingly difficult for them to have it explained to them that this decision was dumb, because their mental model just doesn’t represent that part of reality. And in a functional organization, there will be communication channels whereby someone at the ground level can pull the red handle and say: this has got to stop. Something has happened which doesn’t fit into the model, and everything is going to fall apart.

Lots of organizations do not invest time and resources in creating and maintaining those communication channels. And those organizations will only find out that something’s gone wrong when it’s far too late to do anything about it.

Danny: This seems to relate to John Boyd’s idea of the OODA loop, right? Observe, orient, decide, act. And I wonder — this can be a sort of inadvertent effect of the world just being more complex than the organization can handle, or it can be a strategy by an adversary who tries to intentionally overload someone’s processing capacity. Which, of course, happens a lot.

Dan: Yeah. I think that’s the thing that connects the two books I’ve written so far. Because the ultimate adversarial context is fraud. And fraud works by presenting information that can’t be represented, because it’s been manufactured in order to look like something it isn’t. And that’s what I’m working on in my current book, in a number of other contexts — that adversarial contexts are the worst possible information environment.

Because whenever information has to pass over a boundary between two people or two organizations, there is loss. These are easy theorems from Claude Shannon which are very hard to represent in economics. And resources have to be expended to make sure that that transmission happens, and to effectively overcome that loss. When the boundary is adversarial, the information loss is almost total, in my view. People who are in an adversarial relationship find it incredibly difficult to communicate with each other.

So if you and me were engaged in litigation at the moment, this would be an absolutely painful podcast for your listeners, because everything that we would say, we would be thinking how it might be used against us. And the amount of communication would just shrink down to that. And the limiting case of an adversarial context is something like fraud, where things have got so bad that there’s actually a negative amount of information being transmitted — that once you’ve received that message, you are in a worse relationship with your environment than you were before. And it’s no surprise that a lot of these concepts come from the military.

The whole of information theory and cybernetics has a very military background, because a lot of this work came out of war work in designing secure communication systems, in designing automated gun sights, and in designing other systems that were meant to be robust to shocks coming in from their environments. And we’re just applying that to wider questions of bigger systems that are also trying to remain viable and robust in the face of shocks from their environment.

Danny: Can you say more about the new book that you’re working on? You’ve hinted at it twice.

Dan: Yeah, sure. The title of the book is The Problem Factory. It’s about how institutions and organizations get in each other’s way — and get in their own way. If people want a preview of some of the things I’ve been talking about, I wrote a working paper for the Niskanen Center at Johns Hopkins University which was also called The Problem Factory. So for the next couple of months, if you search for the phrase The Problem Factory and my name, it’ll probably pick that up before the publicity for the new book gets working.

And I was writing about that in the context of planning permission, abundance, and infrastructure provision — where it really does seem to me that most of the problems that we have are manufactured objects. That there are people whose job it is, effectively — or the job that they’ve given themselves — to create problems for other people in the system. And I just got really interested in this concept of adversarial relationships and how they affect society’s ability to process information.

Danny: One thing I was wondering as I was reading your latest book, The Unaccountability Machine, is: how much of the failure that you describe is genuinely a consequence of increased complexity, which is what we’ve been talking about, versus how much of it is just the fact that it’s more visible now? This is Martin Gurri’s thesis — he wrote a great book, The Revolt of the Public, and his thesis essentially is: our institutions have always been disappointing, but what social media has done is just made it painfully visible. And so it’s made it harder to pretend that it isn’t so.

Dan: I think there is an interesting relationship between transparency and complexity, because the two things kind of go together. And not just social media, by the way — if we go back to the 1990s and 2000s, we had people like Tony Blair and Alastair Campbell already complaining that the system was becoming ungovernable because they were under too much pressure from the media. What social media is great at doing is providing people with the illusion that they understand something. Once more, it’s providing another layer of representation, which doesn’t in any way contain the actual complexity of what’s happening, but which is exactly sized to look like it’s comprehensible by an individual human being. And the social media frameworks — effectively, the way that the algorithm and your social media feed and your own actions interact — are such as to tailor that to something which you believe that you can understand.

Because otherwise you keep on interacting with the system until you’ve reached that kind of equilibrium. Which, it turns out, is kind of addictive to people, because it’s that really addictive sense that you understand that part of the world that you’re looking at. The problem is — and this is another one of those great theorems — the model that you are looking at doesn’t actually represent reality. And the model regularly drifts out of contact with reality, because the model is not actually sufficient to represent it. And as a result, you’re always being hit by things which aren’t consistent with your model of reality. And that’s always putting you into that kind of panic state that we were talking about earlier, where you’re having to throw out pieces of information, you’re having to rewire your setup.

And so the visibility and transparency is part of it. But my view is that what’s actually causing the problems is that that transparency is putting people into this constant panic state, because they’re being asked to understand something which it is just not possible for any one human being to understand. And this sort of stuff was predicted back in the 1970s. I have a little reading club with a friend of mine where we read books published fifty years ago, in order to see what people were thinking fifty years ago and to see how the world has developed. And last year was a great year, because the two big books we were interested in that came out in 1975 were Fred Brooks’s The Mythical Man-Month —

Danny: Great book.

Dan: An absolute classic of the computer engineering literature, all about how organizations get more complicated as they grow. But one of the other books that came out in 1975 was Future Shock by Alvin Toffler. A lot of it is very dated 1970s futurism. But the central thesis of that book was that the world was becoming too complicated for human beings to understand, that the media was representing an oversimplified and overly shocking version, and that this was putting people under unbearable stress. And you have to look back at that and say: well, where’s the lie? He didn’t get everything right.

But on the big picture, that book actually turned out to be surprisingly prescient over a fifty-year time horizon.

Danny: And we’ll get to a prediction of yours in a minute. But speaking of models drifting: what did James C. Scott get wrong?

Dan: I don’t think James Scott got much wrong — except the overall framework of “and therefore, anarchism”, which is always there in the background of all of his books. I wrote a sort of mini-obituary for him when he sadly died, I think last year or the year before last, saying that if that guy had turned left when he came out of university rather than turning right, he could have been a great management consultant — specifically, he could have been a great cybernetician. Because, as with all these people, he fundamentally, I think, put his finger on the central problem of governance, which is that when you centralize things, you have to simplify them in order to get them to a level of complexity that the center can understand. And when you simplify things, you are always throwing out information — and some of it will always turn out to be important. I think James Scott was too pessimistic about the flexibility of states and their ability to reorganize.

I think — as almost everyone does — he underestimated the flexibility of industrial society and its ability to reorganize and adapt to conditions. And I think, probably, in some of his books — not so much Seeing Like a State, that’s a stone-cold classic, but one of his other books, the title of which has slipped my mind, the one about hill people and agriculture — he goes a little bit too far towards claiming that this information is intrinsically inaccessible to any level of systematic management. And I don’t think that’s something that you can support.

But by and large, if I got as many of the big calls right as James Scott did, I would be very happy with my career.

Danny: Another one of the greats who I think relates very much to your writing is Charles Perrow, who wrote Normal Accidents — the idea that if you have a system that’s complex, tightly coupled, and has catastrophic potential, then it probably will go wrong. It struck me as quite a cybernetic idea, though I don’t think he quite uses that terminology.

Dan: He doesn’t. But it’s one of the great things about management cybernetics that, left to themselves, intelligent engineers will always reinvent about half of management cybernetics if they’re put in charge of something. I’m reading Perrow’s book at the moment — it’s another absolute classic. The thing is that almost all of that book is about situations in which an important piece of information is missed, simply because it has to be missed — because there is no way that you can bring all that complexity back into the center.

But what Perrow is also very good at saying is that the fact that these things are inevitable doesn’t mean that you should tolerate them, because there are better and worse ways to design the whole system. There are ways that will process that information efficiently. One thing that I think actually takes that kind of model a bit further is Gill Kernick’s book on the Grenfell Tower fire, which has come out in a second edition this year. And she, I think, does better than anyone else so far in bringing together the safety engineering school — I think her original career was in oil rig safety engineering — and bringing that into the same space as politics and governance and popular consent, and bringing those two things together. It’s a very powerful book, and I absolutely recommend it.

Danny: I think what you said before is really a key phrase: better rather than worse. And I think this also comes out in The Unaccountability Machine, where — if I read you right — if we want to run economies at the scale that we’re doing, a certain level of abstraction is unavoidable. A certain level of oops is also probably unavoidable — there’s just nothing we can do about it. But what we can do is make sure we do better rather than worse on these things.

Dan: Absolutely. And I think that’s one point where I disagree with Charles Perrow a bit. That book is very much of its time — he does a lot more in there about recombinant DNA than anyone would write about today, because it looked like it was going to be a much bigger technology than it was when he was writing it. Perrow, towards the end of that book, starts saying that if you do everything you can and minimize the possibility of system accidents, it might still be that the risk is too big to be tolerated, and therefore you just aren’t allowed to do the thing. And he says that there are some technologies and activities which might be intrinsically so dangerous that they shouldn’t be attempted, because there’s no way of safely engineering them to a suitable level.

At various points he comes close to saying that about nuclear energy. I don’t think he ever actually says it about nuclear energy, but the structure is there for anyone who wants to use it. And now we’re getting people saying basically the same thing about AI. We certainly are getting people — even people who are very much on the cutting edge of that technology — saying that they don’t think it can be made safe, and that the whole project needs to be stopped or limited at some stage. I personally am not sure of that. I don’t think that is necessarily as well supported by the science as some of the AI safety people say.

It seems to me that — and this is what I’m working on at the moment — every time you’re building capability for problems, you’re building capability for solutions, because it’s the same capability. Unfortunately, that does mean vice versa: every time you’re building capability for solutions, you’re building capability for people who want to cause problems. But I think these governance problems can be tackled, and it’s a bit of a counsel of despair to say that we should give up on anything, really, because we can’t make it better until it reaches a level that’s acceptable to us.

Danny: Well, I think the counter to that would be people saying: yes, but that is fine for problems at which we get more than one shot. If you genuinely believe that this is a problem at which you’re afforded exactly one shot and one shot only, then I think it would logically follow that we should be quite cautious about it. So I think everything essentially turns on that — whether you believe it or not.

Dan: Yeah — fair point, fair point. I think that’s a case for being cautious, rather than a case for saying that it’s completely impossible. But yes, broadly speaking, fair point. I don’t think we would actually differ on questions of what should actually be done there.

Danny: Which prediction are you most proud of?

Dan: The one thing where I will actually say that I nailed it, in writing, in an economically useful amount of time, was that I wrote a research note in — I think it was late 2006, early 2007 — called Even in Ireland, Trees Don’t Grow to the Sky. In which I was just able to plot the Irish property expansion, point to the point at which it had clearly become a bubble — because the cash flow on the property was no longer covering the debt — and predict when it would come to an end, when the Irish banks exhausted their borrowing capacity. And I think I got that one pretty much bob on. I made a certain amount of money out of it — not as much as I should have done, because I changed jobs and Credit Suisse proprietary trading was limited. But the Irish property bubble is the one where it just all fell into place. It’s a great feeling when it all falls into place and you subjectively know that you are right. I was more confident in that than in any other prediction I’ve ever made in my life.

I think that would be my favorite.

Danny: That’s a pretty good one to nail. What should we all know about Brompton bikes?

Dan: Oh, Brompton bikes — great products. You’re asking because I wrote that book. I wrote a book about the Brompton company — co-wrote, actually, with Will Butler-Adams, who’s the CEO. And a lot of my work there was just to sit down and hear him brain-dump, and then write it up and structure it into a book. And it’s an incredibly privileged position, to actually be able to sit down with a chief executive — I’ve got hours and hours of recordings and transcripts — and hear someone who understands their industry absolutely perfectly talk to you about how they go about doing things.

The thing about Brompton bikes is that everything from that bike comes out from the hinge of the bicycle. Because if you think about the one thing that you shouldn’t do when you’re designing a bicycle, it’s to put a great big hinge in the middle of the frame. And once you’ve made that compromise, your ability to make any other compromises with any of the other components is extremely small. And from that central fact, the entire organizational history of the Brompton Bicycle Company — since the 1970s; it’s been going for nearly fifty years now — basically comes. Because everything else they do, they have to do it perfect. They can’t stand any other compromises, because they’ve already put that huge hinge in the middle of the frame.

And that’s their budget for cutting corners and making trade-offs all gone, right there. But yeah, it’s a great company. I saw today that they’ve got a bunch of new investment, and so I’m hoping that they are going to be able to build their new factory and really do some exciting things.

Danny: Brilliant. I’ve been spending quite a bit of time recently running scenario exercises. You wrote about the geopolitical risk reverse stress test run by the European Central Bank, the ECB, and you said you quite liked it. I think the format is worth explaining, because it’s quite unusual. So what did they do, and what did you like about it?

Dan: It’s really funny — they’re going to be publishing the results in a couple of months. And what they’ve told all of the banks they cover is that, rather than them giving the banks a scenario and asking them to say what the impact would be, they say: well, we’re going to give you an outcome — which is to say that you lost 300 basis points of your capital ratio through some geopolitical risk event — and you have to make up the scenario that delivers that. So you have to tell some kind of geopolitical story which would cost you that much of your capital.

And I wrote about this for FT Alphaville a while back. I’m not sure how some of them are going to do that. Because if you think about a bank like BNP Paribas, 300 basis points of their capital base is tens of billions of euros. So they’re actually asking: come up with a situation in which you lose twenty billion euros. And I think it is very, very unlikely that the closure of the Strait of Hormuz would cost BNP Paribas even a single billion euros.

So they’re going to be there in a room, having to say: well, what’s twenty times as bad as war with Iran? And they will come up with something. They will come up with some kind of cyber-attack scenario, or an expansion of the Russia–Ukraine war, or something. But the thing I like about it is that it’s actually bringing scenario planning back to its original purpose, which is to actually think about what could happen and how you would react to it, and to expand that mental model. I’m hoping to start working on this in the next few years, because one of the interesting things about the whole field of scenario analysis is that it caught on like wildfire after Shell used scenario analysis and happened to absolutely nail the Middle East oil crisis of the 1970s.

Danny: It did make them look like geniuses.

Dan: Yeah, absolutely. And they grabbed huge amounts of market share, because they were the only people ready for a world in which the oil price did that and in which the oil supply did that. And unfortunately, it was like Prince’s Purple Rain: it was the absolute triumph which made people think that they could do anything, and it set in place a load of really disappointing subsequent albums.

Because everyone then went into scenario analysis wanting to hit a home run like Shell predicting the Middle East oil crisis of the 1970s, when that’s not actually the purpose. The purpose is, firstly, to start expanding the mental model. And then, in the case of banks, it is to test whether you have the systems that can represent that. Because I’ve been talking in quite abstract terms about the ability of the corporate center, or the ability of the planning authority, to represent reality at the correct level of detail. But this is where the rubber really meets the road, in these kinds of exercises.

The European Central Bank has been on at the banking sector over climate stress tests for the last couple of years, because the ECB asks the banks: could you do a stress test for a series of massive floods, or a series of wildfires? And the banks have to admit that they don’t really have a flood map which covers the locations of all their real estate collateral, or they don’t actually have any information about which of their clients are in areas that are vulnerable to wildfires. And so the purpose of the exercises is always to tease out those kinds of issues, whereby the supervisor can say: well, if you actually can’t represent that economically important possibility in your management systems, how are you managing your risk? How are you managing the spend? Are you — concrete example — throwing away a piece of information that could prove to be very important?

And so I kind of like it that they’re doing something a little bit off the wall, and they’re requiring the risk management departments to get out of their comfort zone. Because you and I know it’s very easy, in a risk management function, to just get into a place where you think that your job is to run Monte Carlo simulations, or you think your job is to maintain spreadsheets — or, what it is now, Python models, whatever — which is probably going to get almost worse as machine learning starts taking over. And then everyone sits around a meeting once every quarter in the risk management department and says: of course, the real risks are the big business things that we’re not thinking about. And everyone goes: oh yes, yes, very important point. We should make a note of that in the minutes.

Danny: We should put that in the spreadsheet.

Dan: Yeah. And then the meeting ends, and everyone goes: but on the other hand, I’ve got to get six spreadsheets out by the end of the week — and everyone goes back to that. So I think it’s the supervisor’s job to kick people out of their comfort zone a little bit. And making them write a little piece of science fiction about how the world went crazy and they lost twenty billion euros — it’s an interesting exercise. It makes the job more interesting.

It makes people think a little bit outside the box. And anything which encourages that, I think, has got to be considered a good thing.

Danny: And what you’ve also pointed out, which I think is really important, is that it also gets you out of the problem of having to specify the scenario — which, if the regulator did it, they would get unending amounts of flak for, of course. And this just elegantly lets the regulated entity figure out the answer.

Dan: Oh yeah. The European Central Bank having to come up with an actual geopolitical risk scenario — can you imagine Christine Lagarde at a press conference where someone says: what, so you think that Putin’s going to invade Poland? — having to answer that. Yeah, it’s nice.

And also, there’s no way you can game it. You’ve specified what the result is going to be, so there is no point in anyone playing around with their systems in order to deliver a nicer answer — which is the other great endemic problem in risk management systems: people always tweaking the model to get the answer that they wanted.

Danny: Right. With that, Dan: thank you so much. This was a great pleasure. Thanks for coming on.

Dan: No — thank you very much for having me on.

Danny: Thanks for listening to High Variance. You can subscribe to this podcast on Apple Podcasts, Spotify, or wherever you get your podcasts. If you like this podcast, please give us a rating and leave a review. This makes a big difference, particularly for newer podcasts like this one.