2.8.1 Which score do you think is more important?

Course subject(s) Module 2. Calibration and Information score

I D E A L” by slimmer_jimmer is licensed under CC BY-NC-ND 2.0

You have learned that the assessments of experts can be evaluated with two objective measures, that is the calibration score (or the statistical accuracy) and the information score.

You have also learned that the higher both scores are, the better. So ideally, an expert would have a high calibration score as well as a high information score. Right? That would be ideal!

Unfortunately we cannot always find the ideal expert, that has both objective qualities, that is both highly statistical accurate and highly informative. OK, if both qualities cannot be achieved, the next good thing is to have an expert with one of the qualities.

Assume you could choose between these qualities. Which one would you choose?

Let’s consider again our example of the Dutch eating habits. Recall the assessments of Expert 1 and Expert 2.

Question Realization Expert 1

5%   50%  95%

Expert 2

5%   50%  95%

Expert 3

5%   50%  95%

1 50 44    46    49 30    40    55 38    47    55
2 7 9    12    15 1    15    20 2     8     17
3 108 102    106    110 60    80    95 91    99    106
4 66 55    59    64 53    70    80 58   68    75
5 24 28    31    35 10    19    30 26    35    43

Calibration score versus information score

– Expert 1 assessments lead to a high information score (1.18), but a very low statistical accuracy (0.002)
– while Expert 2 assessments are statistically quite accurate (0.61) but less informative (0.18).

Creative Commons License
Decision Making Under Uncertainty: Introduction to Structured Expert Judgment by TU Delft OpenCourseWare is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at https://online-learning.tudelft.nl/courses/decision-making-under-uncertainty-introduction-to-structured-expert-judgment//.
Back to top