So, what we saw with the last exercise was that removing an expert which has a low calibration score might actually decrease the performance of the PWDM.

This is, of course, not always the case!

But can we find a subset of experts for which the combined score of the Decision Maker is the highest possible?

What are the possible subsets?
– all three experts
– two experts and
– one expert

The selection of removing experts is done using the calibration score criterion: the expert with the lowest calibration score will be chosen to be removed.

Well, let’s see that for our example!

The options are then
– Expert A, Expert B and Expert C ==>PWDM3 (PWDM computed using all 3 experts)
– Expert B and Expert C ==> PWDM2 (PWDM computed using 2 experts)
– Expert B (since it’s the highest calibrated expert!) ==> PWDM1 (PWDM computed using 1 expert)

Recall the calibration and the information scores of the PWDM and EWDM when taking into account Expert A and without Expert A assessments (PWDM_updated).

In considering PWDM1, we look for the expert with the highest calibration score in the initial pool of experts. For this, consider all experts’ calibration scores along with their information scores.

Which one has the highest combined score?