Tutorial Calibration
Contents |
Motivation
In order to apply hydrological models successfully it is necessary to define model parameters accurately. A direct measurement of the parameters is mostly not possible, too expensive or there is no clear physical relation. For those reasons the parameters are adjusted in a trial and error process in so far that the simulated factors (e.g. runoff) correspond best to the values measured. This task can be time-consuming and difficult if the corresponding model is complex or has a large number of parameters.
Requirements
A basic requirement is, of course, a functioning model.
To evaluate the quality of an individual simulation mostly a singular measurement is used. Frequently used examples are the mean square error or the Nash-Sutcliff efficiency. The Jena Adaptable Modelling System the Standard Efficiency Calculator is available for this reason. The image below shows an example configuration of this component
Bild:Eff calc.jpg
The component calculates the efficiencies listed in parameter effMethod with:
- 1 -> Nash - Sutcliff efficiency with power 1
- 2 -> Nash - Sutcliff efficiency with power 2
- 3 -> Nash - Sutcliff efficiency of the logarithmic values with power 1
- 4 -> Nash - Sutcliff efficiency of the logarithmic values with power 2
- 5 -> Index of Agreement (1)
- 6 -> Index of Agreement (2)
- 7 -> regression coefficient
- 8 -> WR2
- 9 -> DSGRAD
- 10 -> absolute volume error
- 11 -> mean square error
- 12 -> PBIAS
- 13 -> PBIAS2
By using the attributes validation and prediction the simulation and validation data are transferred to the component in order to compare them. The time interval in which both time periods should be compared is determined by the attribute effTimeInterval. The modelTimeInterval shows the modeling time. In most applications those two time spans do not correspond. Many models show bad mappings at the beginning of the modeling process since they often need a certain time until internal conditions become stationary. The time of initialising should normally not be considered. Often the qualities of models in a certain time period are of interest (e.g. during snowmelt), so that several efficiency components can be used which consider different time periods.
Calibration
There are two common methods for calibrating a model with the integrated Standard Efficiency Calculator:
Offline Calibration
First, it is possible to carry out a calibration locally on your own computer. However, this uses computing resources for a longer period of time. Step 1
Open the JAMS User Interface Model Editor (JUICE).
Load the model you want to calibrate.
The following window should appear:
Bild:Juice kalibration 1.jpg
Step 2
Select the entry Kalibrierungswizard under the item Modell.
A dialog should appear which offers the configuration of the calibration (see image).
Bild:optimization_wizard.jpg
Step 3
In the middle of the dialog area on the left hand side the potential parameters of the model are listed. Select those parameters from the list which you want to calibrate. If you want to select more than one parameter, hold down CTRL while selecting the parameters. Now you can see several input fields for each parameter on the right hand side. You can choose the lower and upper bound which define the area in which the parameter can vary. In addition, it is possible to define a start value. This makes sense if a parameter is already known which suits as a starting point for the calibration.
Step 4
In the the lower area in the dialog the target function is specified. Either a single criterion or various criteria can be selected (for more than one hold CTRL). By choosing various criteria a multi-criteria optimization problem is created which differs significantly from a (common) one-criterion optimization problem regarding its solution. Since not every optimization method is suitable for a multi-criteria problems either, not all optimizers are at disposal this time.
For every selected optimization criterion a list box appears on the right hand side of this section. It determines whether the criterion will be minimized, maximized, absolutely minimized (i.e. min |f(x)| e.g. for the absolute volume error) or absolutely maximized.
Step 5
When parameters and target criteria are selected, it is necessary to choose an optimization method for the calibration. In the upper area of the dialog you can see a list box which lists several optimization methods. At the moment the following methods are available:
- Shuffled Complex Evolution
- Branch and Bound
- Nelder Mead
- DIRECT
- Gutmann Method
- Random Sampling
- Latin Hypercube Random Sampler
- Gauß Process Optimizer
- NSGA-2
- MOCOM
- Paralleled Random Sampler
- Paralleled SCE
For the application with single target criterion the Shuffled Complex Evolution (SCE) and DIRECT are recommended. Most probably both find a global optimized parameterization. It has been attested that DIRECT shows robust operation and no parameterization is necessary. SCE only needs one parameter: the Anzahl der Komplexe (number of complexes). This parameter indirectly controls whether the parameter search area is browsed rather broadly or if the process quickly concentrates on a (sometimes local) minimum. In most cases, the default value 2 can be used. For multi-criteria optimization NSGA - 2 seems to provide very good results. However, there are more detailed analyses to be carried out.
In addition, there are three check boxes available, independent of the selected method.
- irrelevante Komponenten entfernen: deletes components from the model structure which certainly do not influence the calibration criterion
- GUI Komponenten entfernen: deletes all graphical components from the model structure. This option is strongly recommended since e.g. diagrams have a strong negative influence on the calibration.
- Modellstruktur optimieren: at the moment without function
Step 6
In order to finish the configuration, click on the button XML erzeugen. This closes the dialog and a new (modified) model is set up. You can start the calibration by starting the modeling. Please keep in mind that the calibration assistant modifies your output data stores in such a way that for every model run during calibration model parameters and target criteria are given. If you want to avoid this output, delete the file optimization_wizard_optimizer.xml manually.
Online Calibration
As an alternative to offline calibration the process can be carried out on the computing cluster of the Department of Geoinformatics, Hydrology and Modelling at the Friedrich - Schiller - University Jena. This does not occupy any local computing resources and allows a calibration of up to four models at the same time.
Step 1
Open your Internet browser (e.g. Firefox) and go to website [[1]] You will now see the following window and you will require a login. In order to register please contact
- Christian Fischer
- christian.fischer.2@uni-jena.de
Step 2
After the registration you will see the main window of the application.
Bild:Optas_main.png
The window is split up into three parts. In the middle you can see a list of all tasks that have been started. All tasks which have been finished are marked in blue color and tasks which are being carried out are marked green. Red entries indicate that an error has occurred during the configuration of a new task/calibration. The right part of the window shows different contents. When starting the application, you can see news about the OPTAS project. In general, information up to a size of 1 megabyte is shown. Greater data sizes are not shown because it affects the page view. Instead, a zip file is created from the information which is then available for download.
Step 3
In order to carry out a new calibration, click on "Create Optimization Run". You will now see the first step of the calibration process.
Bild:Optas_step1.png
In this step you will be asked to enter the data which is necessary. Load the model for the calibration (jam file) into the first file dialog. Pack your working directory of the model as a zip archive. A suitable programs is for example 7-zip. Please note:
- the working directory cannot be in a subfolder
- please do not load any unnecessary files (e.g. in the output file). The upload is limited to 20mb.
- the file default.jap is not allowed to be in the working directory
Now load the packed working directory into the second file dialog. The modeling will usually be carried out by using the current version of JAMS/J2000. Libraries for J2000g and J2000-S are usually available as well, so you will not need any files for most applications.
Should you need any versions of component libraries which are particularly adapted for models J2000, J2000-S or J2000g, please load them (unpacked) into the corresponding file dialogs. If your models requires additional libraries, you can indicate them as a zip archive in the corresponding file dialog.
To finish, click on Submit. The files that you indicated are now are now being loaded on the server environment.
The management server (sun) checks the host for free capacities and assigns a host to your Task.
Step 4
After transferring the model and the workspace, the following dialog for the selection of parameters and goal criteria appears.
Bild:Optas_step2.png
The first list shows all potential parameters of the model. Select those parameters which you want to calibrate. If you want to select several parameters, hold CTRL when marking them.
In the low part of the dialog a target function is specified. You can either select a single criterion or several criteria (hold CTRL). When selecting several criteria, a multi-criteria optimization problem is created which differs significantly from a (common) one-criteria optimization problem regarding its solution characteristics. Since not every optimization method is suitable for multi-criteria problems, not all optimizers are available in this case.
Step 5
In this step, the valid range of values is to be specified for every parameter. For every selected parameter an input field is available for the specification for upper and lower boundaries. The fields are occupied by default values, if they are available for the specific parameter. These default values are mostly very broad, so it is recommended to carry out further restrictions. In addition, there is an optional possibility to indicate the start value. This is helpful if an adequate set of parameters is already available which is taken as the starting point of the calibration.
In the second part, you can choose for every criterion whether it should be
- minimized (example RSME - root square mean error),
- maximized (Beispiel r²,E1,E2,logE1,logE2),
- absolutely minimized (i.e. min |f(x)| e.g. for absolute volume error),
- or absolutely maximized.
The following image shows the corresponding dialog. In this example, the parameters of the snow module were chosen and the default thresholds were entered. Bild:Optas_step3.png
Step 6
You are now asked to choose the desired optimization method. At the moment, the following methods are available:
- Shuffled Complex Evolution (Duan et al., 1992)
- Branch and Bound (Horst et al., 2000)
- Nelder Mead
- DIRECT (Finkel, D., 2003)
- Gutmann Method (Gutmann .. ?)
- Random Sampling
- Latin Hypercube Random Sampler
- Gaußprozessoptimierer
- NSGA-2 (Deb et al., ?)
- MOCOM ()
- Paralleled random sampling
- Paralleled SCE
For most one-criterion applications the Shuffled Complex Evolution (SCE) method and the DIRECT method can be recommended. For multi-criteria tasks the NSGA-2 mostly delivers excellent results.
Step 7
Many optimizers can be parametrized according to the specific task. SCE has a freely selectable parameter: the number of complexes. A small value leads to rapid convergence, though with higher risk of not identifying the global solution, whereas a greater value increases calculation time significantly while strongly improving the possibility of finding an optimum. Mostly a default value of 2 can be used to work with. In contrast, DIRECT for example does not possess any parameters. It shows a very robust behavior and no parametrization of the method is required. Moreover, three check boxes are available which are not subject to the specific method.
- remove not used components: removes components from the model structure which definitely do not influence the calibration criterion
- remove GUI-components: removes all graphical components from the model structure. This option is highly recommended since e.g. diagrams influence the efficiency of the calibration in a very negative way
- optimize model structure: at the moment without function
The following image shows step 7 for the configuration of the SCE method.
Step 8
Before starting the optimization you can indicate which variables/attributes of the model should appear in the output. As a default, those parameters and target criteria are shown which have already been chosen which makes manual selection unnecessary. Pleas note that saving temporally and spatially varying attributes (z.B. outRD1 für jede HRU) may require a huge amount and this may exceed the size belonging to a user account. Bild:Optas_step6.png
Step 9
You now receive a summary of all modifications which have been carried out automatically. Particularly, you now have a list of components which are classified as irrelevant. By clicking on Submit you start the optimization process.
Operating OPTAS
After having started the calibration of the model, you should now see a new job on the main page. This is shown green to indicate that the execution is not yet completed. You also see the ID, the start date and the estimated date of completion for the optimization. If you select the generated job, you now have various options at your disposal.
Show XML
You can see the model description for carrying out this job which has automatically been generated. As soon as you click on the button, it appears on the right hand side in the window and can be downloaded from there as well.
Show Infolog
Click on this button in order to view status messages of the current model. When using an offline execution of the model this information is written in the file info.log or error.log.
Show Ini
The optimization-Ini file contains all settings applied and summarizes them in abbreviated form. This function is meant for the advanced user or developer, hence there is no further explanation at this place.
Show Workspace
This function lists all files and directories in the workspace. By clicking on the directory/file its content is shown. The modeling results can usually be found in the file /current/ which is shown as an example in the following image.
It is to be noted that file contents can only be represented up to a maximum size of 1 megabyte and they are otherwise zipped in a zip file and made available for download.
Show Graph
Dependencies between different JAMS components of a model can be graphically displayed. This function generates a directed graph the node of which is the number of all components in the model. In this graph, a component A is linked to a component B if A is dependent on B, in the sense that B generates data which are directly required from A. Moreover, all components are shown red which have been deleted during model configuration. This dependency graph allows for an analysis of complex dependency relations, as for example the determination of components which are indirectly dependent on a certain parameter. Such kinds of analyses should, however, not be carried out. The graphical output of this graph is primarily used for visualization.
Stop Execution
Stops the execution of one or more model runs.
Delete Workspace Deletes the complete workspace of the model on the server environment. Please use this function when the optimization is completed and when you have saved the data. This discharges the server environment and accelerates the display of the OPTAS environment.
Modify & Restart This function is addressed to advanced users. Often a restart of a model or calibration task is required without further need of modifications in the configuration. By using the function Modify & Restart the configuration file of the optimization can be directly changed and afterwards the restart of the model can be enforced in a new workspace.
Refresh View Updates the view. Please do not use the update function of your browser (usually F5) since this leads to a re-execution of the last command (e.g. starting the model).
Model Validation
The model validation serves as evidence for the suitability of a model for a certain intended purpose (Bratley et al. 1987). Model validation is an important tool for quality assurance. A model is useful for implementation if it proves to be a reliable representation of real processes (Power 1993). Different key figures and proofs may be required subject to the purpose and type of model in order to validate the model.
In general, there are different quality criteria which play a role for model validation:
- Accuracy/precision: The accuracy of a model represents the deviation of simulated values in comparison to real (measurement) values. This is mostly quantified by the Nash-Sutcliffe efficiency, degree of determination or mean square error.
- Uncertainty: The input values of an environmental model often contain errors and uncertainties. In addition, there is no complete knowledge regarding a natural system, i.e. the modeling has uncertainties. The impact of these values is quantified by the insecurity of the model, i.e. the uncertainty reveals the confidence interval of a model response.
- Ruggedness: The ability of the model to maintain its function if there are (minor) changes in the natural system (e.g. change of land use in a catchment area). This is closely linked to
- Sensitivity: which indicates the influence of (minor) changes in the input data and model parameters on the result
- Validity: is the property of the model to provide correct results for the right reasons, i.e. the explanation for a system which is given by the model is correct.
Die nachfolgenden Methoden wurde von Klemes (1986) für die Validierung von hydrologischen Modellen vorgeschlagen.
Split Sample Validierung
Die verfügbare Menge an Validierungsdaten (z.B. Abflusszeitreihe) wird in zwei Abschnitte geteilt. Zunächst wird der erste Abschnitt für die Kalibrierung des Modells und die zweite für die Validierung verwendet. Anschließend wird die Kalibrierungs- und Validierungsabschnitt getauscht. Daher sollten beide Abschnitte etwa gleich groß sein und eine ausreichend große Datenmenge für eine adäquate Kalibrierung aufweisen. Das Modell kann als akzeptabel eingestuft werden, wenn beide Validierungsresultate ähnlich sind und unter für die Modellanwendung hinreichend gute Eigenschaften aufweisen. Falls die verfügbare Datenmenge nicht ausreichend groß ist für eine 50/50 Teilung, sollte die Teilung derart durchgeführt werden, dass ein Segment für eine sinnvolle Kalibrierung groß genug ist. Der verbleidende Rest wird für die Validierung verwendet. In diesem Fall sollte die Validierung auf zwei verschiedene Weisen durchgeführt werden. Beispielsweise (a) zunächst die ersten 70% der Daten für die Kalibrierung und die verbleibenden 30% für die Validierung (b) anschließend die letzten 70% der Daten für die Kalibrierung und die verbleibenden ersten 30% für die Validierung.
Kreuzvalidierung
right Die Kreuzvalidierung ist eine Weiterentwicklung des Split-Sample Tests. Dabei wird die verfügbare Datenmenge in n gleich große Abschnitte aufgeteilt. Nun werden jeweils n − 1 Abschnitte für die Kalibrierung verwendet und der verbleibende Abschnitt für die Validierung. Dabei wird der zu validierende Abschnitt durchgetauscht, so dass jeder Datensatz einmal validiert wurde. Diese Methode ist deutlich rechenaufwändiger als die Split Sample Validierung, hat aber große Vorteile, wenn die Menge der verfügbaren Daten gering ist und bietet die Möglichkeit jeden Datensatz zu validieren. Ein Spezialfall dieser Validierung ist die Leave-One-Out Validierung (LOO). Hierbei wird stets genau ein Datensatz für die Validierung verwendet und die restlichen für die Validierung. Auf Grund des enormen Aufwandes ist diese Art der Validierung meist nur in Spezialfällen möglich.
Proxy Basin Test
Hierbei handelt es sich um einen grundlegenden Test für räumliche Übertragbarkeit eines Modells. Falls beispielsweise der Abfluss in einem nicht bemessenen Gebiet C simuliert werden soll, können zwei bemessene Gebiete A und B innerhalb der Region gewählt werden. Das Modell sollte mit Daten aus Gebiet A kalibriert und mit Daten des Gebiets B validiert werden. Anschließend sollten die Rollen der Gebiete A und B getauscht werden. Nur wenn beide Validierungsergebnisse unter Hinblick der Modellierungsanwendung akzeptabel sind, kann die grundlegende Annahme getroffen werden, dass das Modell in der Lage ist den Abfluss in Gebiet C zu simulieren. Dieser Test sollte auch verwendet werden, wenn ein verfügbarer Datensatz in Gebiet C vorhanden ist, aber für einen Split-Sample Test nicht ausreicht. Der vorhandene Datensatz in Gebiet C würde also nicht für die Modellentwicklung verwendet werden, sondern lediglich als zusätzliche Validierung.
Differential Split-Sample Test
Dieser Test sollte immer dann verwendet werden, wenn ein Modell in einem bemessenen Gebiet unter neuen Bedingungen verwendet werden soll. Dieser Test kann unterschiedliche Varianten aufweisen, abhängig von der Art wie sich Bedingungen/Eigenschaften des Gebietes verändert haben. Hierzu gehören beispielsweise klimatische Veränderungen, Veränderungen in der Landnutzung, Errichtung von Bauwerken, die die Hydrologie im Gebiet beeinflussen und Ähnliches.
Für die Simulation des Effektes durch den Klimawandel, sollte der Test folgende Form haben. Zwei Zeitperioden mit verschiedenen klimatischen Parametern sollten in den historischen Datensätzen identifiziert werden. Beispielsweise eine Zeitperiode mit hohem durchschnittlichen Niederschlägen und eine weitere mit niedrigen Niederschlägen. Falls das Modell für die Abflusssimulation von feuchten Klimaszenarien verwendet werden soll, sollte das Modell mit dem trockenen Datensatz kalibriert werden und mit die Validierung mit dem "feuchten" Datensatz erfolgen. Falls es für die Simulation eines trockenen Szenarios verwendet werden soll, sollte genau umgekehrt vorgegangen werden. Im Allgemeinen sollte die Validierung das Modell unter veränderten Bedingungen testen. Falls in den historischen Daten keine Abschnitte mit signifikant unterschiedlichen klimatischen Parameter identifiziert werden können, sollte das Modell in einem alternativen Gebiet getestet werden, in dem der Differential Split-Sample Test durchgeführt werden kann. Dies ist meistens der Fall, wenn es sich nicht um klimatische Veränderungen, sondern beispielsweise um Änderungen in der Landnutzung handelt. In diesem Falle sollte ein bemessenes Gebiet gefunden werden, in dem eine ähnliche Landnutzungsänderung stattgefunden hat. Die Kalibrierung sollte in diesem Gebiet mit originaler Landnutzung erfolgen und die Validierung auf den Daten mit neuer Landnutzung.
Wenn alternative Gebiete verwendet werden, sollte mindestes zwei derartige Gebiete identifiziert werden. Das Modell wird für beide Gebiete kalibriert und validiert. Die Validierung ist nur dann erfolgreich, wenn die Ergebnisse für beide Gebiete ähnlich sind. Dabei muss beachtet werden, dass der Differential Split-Sample Test im Gegensatz zum Proxy - Basin - Test für beide Gebiet unabhängig durchgeführt wird.
Proxy-basin Differential Split-Sample Test
Wie bereits der Name suggeriert ist dieser Test die Verbindung aus Proxy-basin und Differential Split-Sample Test und sollte dann angewendet werden, wenn ein Modell räumlich übertragbar sein soll und unter veränderten Bedingungen angewendet wird.
