During this module, you have been introduced to the aggregation of experts’ assessments as well as to the evaluation of the aggregated assessments.

The Classical Model uses the mathematical aggregation of experts assessments, known as the weighted average or linear pooling. As it was mentioned beforehand during the course, other aggregation methods are possible in order to combine experts’ assessment. They generally classify into behavioral aggregation and mixed type of aggregation.

The core idea of the Classical Model is to objectively evaluate experts’ assessments. During this module, you have learned that this evaluation can also be transferred to Decision Makers. The idea of evaluating DMs with respect to different measures was driven by the question:

Which weights should be used in averaging experts’ assessments?

Finally, quantile aggregation can be viewed as an alternative to the aggregation of experts’ distributions. Is this a better approach?

Here are some references to all these topics and answers to all these questions. Enjoy!

Want to learn even more? Please feel to contact the course team!

“reading” by Terry Freedman is licensed under CC BY-NC-ND 2.0

Evaluation of DMs performance

• Cooke, R. M., & Goossens, L. L. (2008). TU Delft expert judgment data base. Reliability Engineering & System Safety, 93(5), 657-674.

A copy of the first author can read here. (The copy is provided for non-commercial and education use.)

• Colson, A. R., & Cooke, R. M. (2017). Cross validation for the classical model of structured expert judgment. Reliability Engineering & System Safety, 163, 109-120.

Equal weights and averaging quantiles

• Colson, A. R., & Cooke, R. M. (2017). Cross validation for the classical model of structured expert judgment. Reliability Engineering & System Safety, 163, 109-120.
• Cooke, R. M. (2018). Validation in the classical model. In Elicitation (pp. 37-59). Springer, Cham.
• Morgan, M. G. (2014). Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National academy of Sciences, 111(20), 7176-7184.

Behavioral aggregation

• O’Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., … & Rakow, T. (2006). Uncertain judgements: eliciting experts’ probabilities. John Wiley & Sons.
• A very recent paper of O’Hagan can be found here.

Mixed methods

• The best known mixed method is the Delphi method. There are numerous sources to read about the method, and here is just one.
• The mixed method that you will learn during this course is called the IDEA protocol. Just a couple of references can be found here and here.