“question mark” by Damián Navas is licensed under CC BY-NC-ND 2.0

You might wonder now:  why do we ask experts questions to which we already know the answer to?

This is because we want to objectively evaluate experts’ assessments. We can do that by asking them questions to which we know the answers to, which are referred to as calibration or seed questions (or variables).

In contrast, the questions that drive the expert judgment study are called questions (or variables) of interest.

You might have heard the expression “expert elicitation” since the course started. It refers to the process of asking and obtaining expert assessments. This process is carried in a structured manner, and we will discuss the practical details later in the course.

What distinguishes the Classical Model from many other methods of performing expert judgment is that it includes two types of questions: the questions of interest and the calibration (seed) questions.

“question mark” by pineapple15 is licensed under CC BY-NC-SA 2.0

Calibration questions

The calibration questions (or seed variables) are questions whose answers are known to the analyst, but should not be known by the experts. The questions should regard an uncertainty quantity, for which experts can provide their assessments.

Finding good calibration questions is a crucial step in any expert judgment study. Usually, data coming from official, yet not public reports or data. The ideal scenario is when the answers to the calibration questions are known soon after the elicitation of the experts. Reports can be released or data become available. Unfortunately this is not always possible.

“question mark” by pineapple15 is licensed under CC BY-NC-SA 2.0

Questions of interest

To differentiate between the calibration questions and the questions that drive the entire structured expert judgment elicitation, we refer to the latter as questions of interest. Defining the questions of interest is usually the starting point of any study.

“question mark” by Damián Navas is licensed under CC BY-NC-ND 2.0