3.7.4 The information score of the DM
Course subject(s)
Module 3. Performance-based weights and the Decision Maker
Along with the calibration score, the information score can also be computed for the two considered Decision Makers.
It is not time to investigate the informativeness of the two Decision Makers.
Consider the previous example about the correlation between intelligence in primary school children and other personality traits and success in several aspects of life.
Recall the experts’ calibration and information scores (up to the third decimal).
| Calibration score | Information score | |
|---|---|---|
| Expert A | 3.621E-005 | 1.418 |
| Expert B | 0.608 | 0.558 |
| Expert C | 0.327 | 1.354 |
These scores lead to the following performance-based weights:
0.00006565341 (Expert A), 0.4338007 (Expert B) and 0.5661336 (Expert C).
In turn, the performance-based weights lead to the following assessments for the Performance-based Decision Maker (PWDM) and the Equal Weight Decision Maker (EWDM). Recall that the PWDM and EWDM assessments are obtained from averaging distributions rather than averaging quantiles.
| Question | Realization | Expert A
5% 50% 95% |
Expert B
5% 50% 95% |
Expert C
5% 50% 95% |
PWDM
5% 50% 95% |
EWDM
5% 50% 95% |
|---|---|---|---|---|---|---|
| 1 | 24 | 2 5 7 | 20 23 34 | 20 25 28 | 19.97 24.48 33.11 | 2.55 21.77 32.39 |
| 2 | 11 | 12 13 15 | 8 10 11 | 10 13 15 | 8.19 10.91 14.86 | 8.31 12.23 14.91 |
| 3 | 68 | 87 89 92 | 60 70 90 | 65 67 70 | 60.75 67.56 87.94 | 61.43 71.08 91.61 |
| 4 | 47 | 2 5 8 | 20 35 45 | 42 44 46 | 21.02 42.8 45.97 | 2.55 34.28 45.78 |
| 5 | 31 | 48 50 52 | 30 35 40 | 32 33 35 | 30.29 33.47 39.82 | 30.66 35.34 51.63 |
Recall that the information score is computed per question.
Then an average information score will determine how informative each expert, but also each Decision Maker is.
Let’s see how informative PWDM and EWDM are for this example!
They show that performance-based weighting CAN lead to better information scores and/or calibration scores than equal weighting of experts’ opinion!
We will see in the following modules that this conclusion is not resulting from fictional exercises only, but from numerous real case studies!
Accounting for the performance of experts’ assessments in assessing uncertainty does lead to an improved performance of the combined assessments!
Decision Making Under Uncertainty: Introduction to Structured Expert Judgment by TU Delft OpenCourseWare is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at https://online-learning.tudelft.nl/courses/decision-making-under-uncertainty-introduction-to-structured-expert-judgment//.