By now, most modelers are aware that the calibration process does not normally result in a unique parameter set. In fact, in many instances of model calibration, there are many different parameter sets which either calibrate, or almost calibrate, the model due to the fact that parameters are often highly correlated with each other. Yet in most modeling situations, model predictions are made with just one set of parameters! The question that is often asked, but rarely answered, is "what would model predictions have been if another set of parameters, which also calibrated (or nearly calibrated) the model were employed in the predictive process." Or, even more importantly, "what is the worst (or best) prediction that it is possible to make with a parameter set that calibrates the model". For nonlinear models (i.e. most models) this is a very difficult question to answer. However it can now be answered using PEST's predictive analyzer.

A series of illustrations shows how it works. Figure 1 shows contours of the "objective function" in two-parameter space. For PEST, the objective function is the sum of squared deviations between model outcomes and corresponding field data. The lower it is, the better is the model calibrated. In most instances, the region of "allowed parameter space" where the objective function is low enough for the model to be considered as calibrated, is long and skinny as is shown in the figure. Any parameter set within the shaded region of Figure 1 can be considered to calibrate the model. Note that it is not only calibration conditions which enforce constraints on parameter values; in most cases knowledge and physical constraints result in the imposition of realistic bounds on parameter values as well. These bounds are also shown in Figure 1.

Figure 1. Contours of the objective function in parameter space. "Allowed parameter space" shown shaded.

Figure 2 shows the dependence of a key model prediction on parameters p1 and p2. The contours increase in value toward the top right of the figure. Thus the higher are p1 and p2 the higher is the model prediction.

Figure 2. Contours of a key model prediction in parameter space.

In many cases of model deployment, it is critical to know the worst (or best) prediction that it is possible to make with parameters that still calibrate the model. The "critical point" illustrated in Figure 3 identifies those values of parameters p1 and p2 that provide this worst (or best) prediction, while still satisfying calibration and knowledge constraints. When run in "predictive analysis mode" PEST finds this critical point.

Figure 3. The critical point in parameter space. It is the goal of predictive analysis to find this point and the model prediction arising from it.

Based on an adaptation of the extremely robust PEST parameter estimation algorithm to the methodology developed in Vecchia and Cooley (1987), PEST will find the critical point in parameter space using an iterative solution procedure, starting from initial parameter estimates that can lie either inside "allowed parameter space"….

Figure 4. Initial parameter estimates that satisfy calibration constraints.

or way outside "allowed parameter space"…

Figure 5. Initial parameter estimates that do not satisfy calibration constraints.

PEST can undertake predictive analysis for a problem involving any number of parameters. The interface between PEST and the model is the same as for the normal PEST, i.e. through the model's own input and output files. Thus the cornerstone of PEST's model independence is preserved.

Actually, PEST allows modelers to undertake predictive analysis using another method as well, i.e. the method of "dual calibration". This involves simultaneous calibration of a model under both calibration and predictive conditions, where a worst (or best) case prediction is used as an "observation" in the parameter estimation process together with the normal calibration dataset. By judiciously choosing observation weights, the user can find a parameter set that achieves a suitably bad (or good) prediction while still keeping the model calibrated.

Reference

Vecchia, A.V. and Cooley, R.L., 1987. Simultaneous Confidence and Prediction Intervals for Nonlinear Regression Models with Application to a Groundwater Flow Model. Water Resources Research, vol23, no. 7, pp1237-1250.