To be a good Bayesian, you have to be well calibrated, meaning that of the events you assign X% probability about X% of them should turn occur. You can use a calibration test to assess a person’s calibration. A calibration test consists of a relatively long list of independent trivia-like questions. The test-taker gives an answer and confidence in their answer (probability of their answer being right). When the test is finished, the results are compiled into a calibration curve showing how the actual % right varied with % confidence (see fig).

As you can see, the chart makes it easy to tell how well calibrated you are; most people in most situations are overconfident (green line).

When I first read about calibration tests (6 months or so ago), I grew excited about taking such tests online. I thought that if I could see my own calibration, I could fix it. Unfortunately, I couldn’t find anything online. I no longer think that calibrating yourself is very easy; it seems extremely difficult, but now Alex Loddengaard and I have created a website where people can take calibration tests: Calibrated Probability Assessment.org. I should note that Tom McCabe also recently created a set of calibration tests.

Currently, I have two tests on the website. They are both, ok, but need work. I have also written up a little bit of information on calibration literature, but this also needs work. I hope to improve the website over time, making it more useful. Anyone who wants to help out, creating test questions, explanations or website building should contact me.

Update:
The main about Calibrationd Probability Assessment so far has been that the test is too long, which makes it fatiguing. Unfortunately, 50 questions is just about the minimum for a test with 5 confidence levels because it takes at least 10 questions to assess 90% confidence.

I think there might be some workaround for this, I could assume a linear relationship, or I could set up a program to do emails of 10 questions a day, but I have to think about how to do these.

Advertisements