Prediction Markets are such a neat technology, clearly we should introduce them into governance, right? Considerations on what make them more likely to be successful.
(I copied this comment from LessWrong, because it will likely be rejected for using the adjective "dogshit" and I refuse to compromise my communication style for whatever fucking hall monitor dipshit has the job of rejecting that)
While I agree completely with you about what you call the "caveats", "loose ends", and "cognitive load" imposed by them, it's ENORMOUSLY frustrating to read this in a public forum. I was already well aware of these problems years ago when I was unexpectedly scouted and hired by Metaculus and given a brief to fix them. But they turned out to be completely, wildly incompetent as an organization, like just dogshit incompetent. They had given contradictory instructions to other people and ultimately just didn't understand what they were even telling people to do, and they were extremely quick to blame their employees for problems without doing any actual study, just going with whoever complained about the other person first or whoever had the better resume. So they ended up firing me within a few months and continuing on to approve these garbage questions.
There is no reason why crowdsourced forecasting oracles actually need to be full of under- and over-specified "gotcha" questions. It's just that there has literally never once been one that was intensively curated by people who weren't idiots. If someone wanted to do that, they could. It is a problem of competence and of these platforms often being created by ideologues who aren't really forecasters at all.
Huh! I hadn't considered whether sufficient moderation could solve the problem. I suspect that it'll be tough if the market ever gets real scale, because the open-endedness will still be present at the heart of the market, and now there'll be an incentive to influence the moderation, but it's worth considering at least.
Incentives to influence the editorial / resolution side only arise when the editors and resolvers make themselves influenceable. Metaculus has a bad case of "client-itis" because most of the people working there have poorly defined responsibilities and are effectively unsupervised, so they end up trying to prove their helpfulness by identifying and pleasing various internal and external "clients".
But you could just not do this.
The amount of work involved in making and resolving these questions is not actually that large. If you imagine a three-man founding team all of whom were real forecasters with common assumptions trying to clone Metaculus in a few months, they could probably make 1,000 high-quality questions and have it be like 10-20% of their total workload at most. And that would be enough that the site could launch and feel heavily populated with questions (perhaps a much better variety and more individually interesting questions than Metaculus or Manifold.) Making new questions on an ongoing basis, resolving old ones, doing moderation etc would probably amount to close to a full time job for one person, as long as we allow that founders typically do like 2 or 3 "full time jobs" worth of actual work in practice, and you typically have 2 or 3 of them. So there is really no need to outsource this function to poorly-supervised "community moderators", make it a free-for-all where each creator "owns" their own question, etc. You can actually keep it close to the vest and be really picky about it, and only let in "the community" once they've been properly indoctrinated.
(This is how Reddit and Wikipedia and all those old-school community platforms worked in their early phases, by the way. They were theoretically open, and they really did mean something by their promise of openness, but the vast majority of the early content was founder-contributed, sometimes behind obscuring pseudonyms!)
Manifold doesn't do this because they're commercially oriented and chasing high engagement, which means pointless nerd-trap questions and people scamming each other. Metaculus failed to do this right because the founders were theoretical physicists who were just as impractical as the stereotypes say they would be. And I don't know enough about the real-money markets to understand why even they seem to struggle with resolutions and specifications, but they do. However I think of these failures less as evidence that the problem is intractable, and more as a signal that the first group to get it right, probably by attending to it from the start as a hard problem, will win a lot of credibility quickly in the forecasting space.
Founders don't scale across time. This could work to create a market that might well last even 10-30 years, but doesn't create an institution that you can depend on for a century. That's not a knock-down argument against doing it, just a reason why it's not how I'm thinking about governance.
(I copied this comment from LessWrong, because it will likely be rejected for using the adjective "dogshit" and I refuse to compromise my communication style for whatever fucking hall monitor dipshit has the job of rejecting that)
While I agree completely with you about what you call the "caveats", "loose ends", and "cognitive load" imposed by them, it's ENORMOUSLY frustrating to read this in a public forum. I was already well aware of these problems years ago when I was unexpectedly scouted and hired by Metaculus and given a brief to fix them. But they turned out to be completely, wildly incompetent as an organization, like just dogshit incompetent. They had given contradictory instructions to other people and ultimately just didn't understand what they were even telling people to do, and they were extremely quick to blame their employees for problems without doing any actual study, just going with whoever complained about the other person first or whoever had the better resume. So they ended up firing me within a few months and continuing on to approve these garbage questions.
There is no reason why crowdsourced forecasting oracles actually need to be full of under- and over-specified "gotcha" questions. It's just that there has literally never once been one that was intensively curated by people who weren't idiots. If someone wanted to do that, they could. It is a problem of competence and of these platforms often being created by ideologues who aren't really forecasters at all.
Huh! I hadn't considered whether sufficient moderation could solve the problem. I suspect that it'll be tough if the market ever gets real scale, because the open-endedness will still be present at the heart of the market, and now there'll be an incentive to influence the moderation, but it's worth considering at least.
Incentives to influence the editorial / resolution side only arise when the editors and resolvers make themselves influenceable. Metaculus has a bad case of "client-itis" because most of the people working there have poorly defined responsibilities and are effectively unsupervised, so they end up trying to prove their helpfulness by identifying and pleasing various internal and external "clients".
But you could just not do this.
The amount of work involved in making and resolving these questions is not actually that large. If you imagine a three-man founding team all of whom were real forecasters with common assumptions trying to clone Metaculus in a few months, they could probably make 1,000 high-quality questions and have it be like 10-20% of their total workload at most. And that would be enough that the site could launch and feel heavily populated with questions (perhaps a much better variety and more individually interesting questions than Metaculus or Manifold.) Making new questions on an ongoing basis, resolving old ones, doing moderation etc would probably amount to close to a full time job for one person, as long as we allow that founders typically do like 2 or 3 "full time jobs" worth of actual work in practice, and you typically have 2 or 3 of them. So there is really no need to outsource this function to poorly-supervised "community moderators", make it a free-for-all where each creator "owns" their own question, etc. You can actually keep it close to the vest and be really picky about it, and only let in "the community" once they've been properly indoctrinated.
(This is how Reddit and Wikipedia and all those old-school community platforms worked in their early phases, by the way. They were theoretically open, and they really did mean something by their promise of openness, but the vast majority of the early content was founder-contributed, sometimes behind obscuring pseudonyms!)
Manifold doesn't do this because they're commercially oriented and chasing high engagement, which means pointless nerd-trap questions and people scamming each other. Metaculus failed to do this right because the founders were theoretical physicists who were just as impractical as the stereotypes say they would be. And I don't know enough about the real-money markets to understand why even they seem to struggle with resolutions and specifications, but they do. However I think of these failures less as evidence that the problem is intractable, and more as a signal that the first group to get it right, probably by attending to it from the start as a hard problem, will win a lot of credibility quickly in the forecasting space.
Founders don't scale across time. This could work to create a market that might well last even 10-30 years, but doesn't create an institution that you can depend on for a century. That's not a knock-down argument against doing it, just a reason why it's not how I'm thinking about governance.