3.7.3 The calibration score of the DM (2)

Course subject(s) Module 3. Performance-based weights and the Decision Maker

Let’s consider another example!

Intelligence and other life aspects

Three experts have been consulted to provide their opinions with respect to the correlation between intelligence in primary school children and other personality traits and success in several aspects of life.

A sample from 10 primary schools was taken in order to obtain realizations, and based on these, the experts can be evaluated in terms of both a calibration and an information score.

“Road to Success – Success – Winning” by ota_photos is licensed under CC BY-SA 2.0

In order to simplify the example, only experts’ quantiles will be presented below.

Five questions were asked, for which the experts provided their assesments, which are provided below.

Question	Realization	Expert A 5% 50% 95%	Expert B 5% 50% 95%	Expert C 5% 50% 95%
1	24	2 5 7	20 23 34	20 25 28
2	11	12 13 15	8 10 11	10 13 15
3	68	87 89 92	60 70 90	65 67 70
4	47	2 5 8	20 35 45	42 44 46
5	31	48 50 52	30 35 40	32 33 35

Again, these assessments are fictional. Real examples would imply more questions, and the calculations by hand would be much harder!

These assessments lead to the following calibration and information scores (up to the third decimal).

You can check the scores if you want!

	Calibration score	Information score
Expert A	3.621E-005	1.418
Expert B	0.608	0.558
Expert C	0.327	1.354

Note that the calibration score of Expert A is very small, that is 0.00003621!

These scores lead to the following performance-based weights:

0.00006565341 (Expert A), 0.4338007 (Expert B) and 0.5661336 (Expert C).

In turn, the performance-based weights lead to the following assessments for the Performance-based Decision Maker (PWDM) and the Equal Weight Decision Maker (EWDM).

Question	Realization	Expert A 5% 50% 95%	Expert B 5% 50% 95%	Expert C 5% 50% 95%	PWDM 5% 50% 95%	EWDM 5% 50% 95%
1	24	2 5 7	20 23 34	20 25 28	19.97 24.48 33.11	2.55 21.77 32.39
2	11	12 13 15	8 10 11	10 13 15	8.19 10.91 14.86	8.31 12.23 14.91
3	68	87 89 92	60 70 90	65 67 70	60.75 67.56 87.94	61.43 71.08 91.61
4	47	2 5 8	20 35 45	42 44 46	21.02 42.8 45.97	2.55 34.28 45.78
5	31	48 50 52	30 35 40	32 33 35	30.29 33.47 39.82	30.66 35.34 51.63

Decision Making Under Uncertainty: Introduction to Structured Expert Judgment by TU Delft OpenCourseWare is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at https://online-learning.tudelft.nl/courses/decision-making-under-uncertainty-introduction-to-structured-expert-judgment//.