Where are the Expert Track Records?

This post on Best Of A Great Lot is a part of a series on the subject of designing a new form of governance. Each piece aims to stand alone, but fits together on the Table of Contents.

Max Read points to Matt Yglesias's description of what makes him successful as the secret to success for content creators:

“It’s the best time there’s ever been to be somebody who can write something coherent quickly,” Yglesias says, over coffee. “I find it relaxing to work. I put things out. People yell at me. I will write again the next day.”

This way of viewing it is about writing a lot, quickly, regularly. But I think the most important part is something else Max said:

In aggregate, over a shorter period of time than you imagine, people forget, or move on, or don’t really care. It is extremely rare to see writers bear any kind of significant career consequences for being wrong, let alone for being cringe.

The term writer here is overloaded. What we're talking about is a public intellectual - someone who makes a living out of having an opinion. Perhaps it’s not a good idea for society to be designed to allow public intellectuals to be so often wrong and bear no consequences.

It gets worse: we don't hold politicians to a track record very well either. Public intellectuals influence our discourse, but politicians make actual society-affecting decisions. After all, let's look at our last few Presidents:

• Biden - Voted for the Iraq War, the War on Drugs, and basically every other mistaken war he had the honor of seeing the Senate approve.

• Trump - Ran on being a businessman, despite his businesses having lost more money than they made over his decades in business. His two greatest successes? Being born into wealth and playing a businessman on reality TV.

• Obama - Obama's best attribute when he ran in 2008 was that he didn't have much track to record, so he could run on pure hope.

• Bush (W) - Started the Iraq War by lying to the American public, then got re-elected. What does it take to get voted out of office, you might start to wonder.

Philip Tetlock’s research suggests experts are worse at predictions than chance.

Most of Tetlock’s questions had three possible answers; the pundits, on average, selected the right answer less than 33 percent of the time. In other words, a dart-throwing chimp would have beaten the vast majority of professionals.

Maybe we should start finding ways to expect our public intellectuals and our politicians to get things right.

Mistakes

Because we don’t keep detailed track records, we pay either too much or too little attention to individual mistakes. I did it above, cherry-picking the strongest reason to be skeptical of each of the last 4 Presidents and ignoring all of their many successes. Every partisan has that opportunity, since there’s simply so much that happens in 4 years of politics. Joe Biden had a decades long career, and I picked less than .004% of the bills he voted on. George W Bush was Governor of Texas and I didn’t even discuss his successes or failures there. Maybe if we added up all of their right and wrong calls we’d have a very different opinion of them.

Mistakes aren’t even always a bad thing. Many business leaders embrace the idea of fail fast because they recognize that making mistakes and learning from them is usually necessary. People who choose to go slow to avoid mistakes often end up avoiding learning. The trick is usually to identify situations where it’s safe(r) to make mistakes and learn from them.

If we were to try to wrap up that idea into a math equation, we might get something like this:

\(Experience = mistakes × learning\)

Are there people who learn things without making mistakes, or who already know enough? Until we have great track records, this seems to be a perennial unresolveable argument.

But for the rest of us, if you’re following the path of making many mistakes and learning from them, we should see one of two things:

• Make fewer mistakes

• Make different mistakes

But we can’t see that by individually evaluating a mistake someone made! We can only see that by looking at the mistakes they made before and the mistakes they’re making now - looking at the Δ either in number or type of mistakes. This kind of visibility is what we should demand of track records.