“I D E A L” by slimmer_jimmer is licensed under CC BY-NC-ND 2.0

You have learned that the assessments of experts can be evaluated with two objective measures, that is the calibration score (or the statistical accuracy) and the information score.

You have also learned that the higher both scores are, the better. So ideally, an expert would have a high calibration score as well as a high information score. Right? That would be ideal!

Unfortunately we cannot always find the ideal expert, that has both objective qualities, that is both highly statistical accurate and highly informative. OK, if both qualities cannot be achieved, the next good thing is to have an expert with one of the qualities.

Assume you could choose between these qualities. Which one would you choose?

Let’s consider again our example of the Dutch eating habits. Recall the assessments of Expert 1 and Expert 2.

Question	Realization	Expert 1 5%   50%  95%	Expert 2 5%   50%  95%	Expert 3 5%   50%  95%
1	50	44    46    49	30    40    55	38    47    55
2	7	9    12    15	1    15    20	2     8     17
3	108	102    106    110	60    80    95	91    99    106
4	66	55    59    64	53    70    80	58   68    75
5	24	28    31    35	10    19    30	26    35    43

Calibration score versus information score

– Expert 1 assessments lead to a high information score (1.18), but a very low statistical accuracy (0.002)
– while Expert 2 assessments are statistically quite accurate (0.61) but less informative (0.18).