Grand visions, detailed plans by experts and gurus with an infallible solution are outdated, says Tim Harford, who writes the immensely popular “The Undercover Economist” column for the Financial Times. In his new book, Adapt: Why Success Always Starts With Failure, Harford says individuals and organisations must continuously improvise rather than obsessively plan. An adaptive trial-and-error method is the only way to tackle issues in a world that has become far too unpredictable and complex for ready-made solutions, argues Harford. We spoke to him to find out how we can all learn to do this.

What is the main context for this book?
The book explores how complex problems are solved, from business innovation, climate change, the war on terror to the financial crisis. While looking at these issues and interviewing the people involved in solving them, I found the same themes emerging again and again: trial and error is a vital part of problem solving in a complex world. We need to experiment and risk failure—this is something we find difficult, both personally and on an institutional level.

Can you recall for us trigger points, inspirations to begin writing?
There were many. One was chatting on a beach with, of all people, the president of Virgin Galactic. I told him I was interested in exploring how innovation prizes work—the Virgin Galactic project having emerged as a result of a famous innovation prize, the Ansari X Prize. He told me that the Spitfire—a crucial fighter aircraft during the Second World War—was also the result of an innovation prize. So I looked into it and the Spitfire became a key part of the book.

On another occasion I wanted to write about a terrible industrial accident that had triggered a small financial crisis in insurance markets. My sister is a safety engineer and she was able to give me a reading list about Piper Alpha, Chernobyl, Bhopal and other industrial disasters. As I studied them—with their mixture of negligence, fraud and sheer bad luck—I came to realise that accidents in these complex, tightly-coupled systems were a very good parallel for the financial crisis. I spoke to several leading safety experts who were convinced that their insights on managing failure could have helped to prevent the financial crisis. It was a serendipitous discovery—but that’s very much in line with the message of the book. A lot of good things happen by accident.

Is this book almost a warning against scale? Are size and adaptability always at odds?
Not entirely. Small organisations are very sensitive to failure so are able to change direction quickly if something is going wrong. Big organisations usually have poor communication between the front line and the top of the organisation. However, big organisations also have the resources and resilience to experiment a lot. Google is a good example of that. I argue that adaptability is difficult for everyone—from governments to individuals—but the reasons why we struggle to experiment and adapt vary depending on circumstance. Sometimes it’s a political problem, sometimes it’s a question of psychology.