The Decision Maker obtain from selecting subsets of experts which has the highest combined score is called the optimized Decision Maker.

To wrap it up, the method to find the Optimized Decision Maker is

1. Remove experts successively with the lowest calibration score
2. Compute the combined score of the corresponding Decision Makers
3. Choose the Decision Maker with the highest combined score.

You have now learned about a method to combine experts’ assessments which result in the best overall performance!

Recall that in the video about the performance based weights, a significance level α was mentioned that can be used to define the performance-based weights.

Cal(e)∗Inf(e)∗1{Cal(e)≥α}

where 1{} denotes the indicator function: it is 1 if the calibration score of the expert is higher than the significance level α and is 0 otherwise.

You can see that setting a certain value for the significance level corresponds to selecting a subset of experts.

Finally, with our example, we can conclude that the optimized Decision Maker is made up of a single expert, that is Expert B.