Possible reasons for Overall Model Test rejection

We learned, so far, that the overall model test statistic is computed as 

\[T=\hat{e}^{\text{T}}Q_{yy}^{-1}\hat{e}.\]

We also learned that the test statistic \(T\) is a realization of the random variable \(\underline{T}\) which follows the central \(\chi^2\)-distribution with \(m-n\) degrees of freedom.  Knowing the distribution, we can define the critical value \(K_\alpha\) corresponding to a certain level of significance (\(\alpha\)), and if  \(T>K_\alpha\) we suspect an error to be present in the mathematical model. The question is now: what if the overall model test results in a rejection? 

To get a better understanding of the possible reasons of the OMT rejection, we rewrite the test statistic as follows:

\[T=\hat{e}^{\text{T}}Q{yy}^{-1}\hat{e}=(y-A\hat{x})^{\text{T}}Q_{yy}^{-1}(y-A\hat{x})\]

By inserting \(\hat{x}=(A^{\text{T}}Q_{yy}^{-1}A)^{-1}A^{\text{T}}Q_{yy}^{-1}y\) in the above equation, we get

\[T=  y^{\text{T}}[I-Q{yy}^{-1}A(A^{\text{T}}Q_{yy}^{-1}A)^{-1}]~Q{yy}^{-1}~[I-(A^{\text{T}}Q_{yy}^{-1}A)^{-1}A^{\text{T}}Q_{yy}^{-1}]y. \]

We can see that the test statistic \(T\) is the function of the following three components:

1. 1. Observation vector \(y\)
2. 2. The functional model defined by the design matrix \(A\) 
3. 3. The stochastic model defined by the covariance matrix \(Q_{yy}\). 

So in general, we can say that the rejection of the overall model test can be possibly due to an error in one of these three components (or a combination of them). With just the OMT, we cannot identify the exact reason for the rejection. The OMT rejection can be caused either by outliers or biases in the observations (i.e., in the vector \(y\)),  a wrong or inappropriate functional model (\(Ax\)), or by an insufficient specification of the observations' uncertainties and the characterisation of random errors in the stochastic model (\(Q_{yy}\)).  Just the overall model test by itself cannot provide the answer.

What to do in case of OMT rejection? 

The overall model test, in fact, checks the overall validity of the assumed mathematical model. In case of rejection, one has to search for a possible model imperfection/misspecification, or in other words, try to identify the model errors which cause the rejection. This identification step involves specification of possible alternative hypotheses regarding the model/data and to test whether an alternative hypothesis seems more likely. The choice of the alternative hypotheses is application dependent and is one of the most difficult tasks in testing theory.  There is no single recipe for the identification step, and it depends very much on one's experience and a-priori knowledge on which types of errors are likely. 

After identification of the possible errors, a proper action/decision should be undertaken/made to adapt the mathematical model accordingly (e.g. by removing the detected outliers, using more appropriate functional model, or scaling the stochastic model)  in a way that the OMT is accepted. 

Note that, whatever model adaptation is applied, we still need to make sure whether the adapted model is valid or not. This means to repeat the OMT again with the new model. 

A special case when we are sure about the functional model

In some situations, it may happen, that we are quite sure about the assumed functional model and also we know a-priori that there is no outlier in the data or it is very unlikely to have an outlier. In these special situations,  one can argue that if the OMT is rejected, the reason is an imperfection in the stochastic model. So the stochastic model or \(Q_{yy}\) should be adapted (e.g., scaled up) accordingly. Instead of using the initial/assumed covariance matrix \(Q_{yy}\), we need to scale up the covariance matrix by a variance factor \(\sigma_{\text{new}}^2\), and use the \(Q_{yy,\text{new}}=\sigma_{\text{new}}^2 Q_{yy}\) instead. It can be proved that an unbiased estimate for  \(\sigma_{\text{new}}^2\) can be given as 

\[\sigma_{\text{new}}^2=\frac{\hat{e}^\text{T}Q_{yy}^{-1}\hat{e} }{m-n}.\]

So in this situation, that the stochastic model is wrong just due to a scaling factor, the estimate obtained from above equation provides the answer to how to scale up the assumed covariance matrix in a way that the OMT is accepted. Note that this approach only works when the \(Q_{yy}\) is wrong up-to a scaling factor. However, if the structure of \(Q_{yy}\) is totally wrong (for example, when we mistakenly assume uncorrelated observations with a diagonal covariance matrix, whereas the observations are correlated and so the correct \(Q_{yy}\) should be fully populated),  the aforementioned scaling-up approach cannot correctly adapt the stochastic model. 

Concluding remark

We saw that the overall model test by itself cannot give the answer to the question of what is the reason for OMT rejection. In other words, OMT is just a tool to detect whether there is a problem/error(s) in the model or not, but it cannot identify the problem/errors. The discussion about different methodologies and the details of the identification and adaptation steps, together with some other quality aspects, such as the concept of reliability, are beyond the scope of this introductory course.

By the above discussion on the possible reasons for the OMT rejection and the outlook on how to adapt the mathematical model,  we conclude this course on observation theory: estimating the unknown.  We leave the more in-depth discussion of identification, adaptation, reliability, and different alternative hypothesis testing to the future courses in this MOOC series on observation theory.