Tutorial Calibration
Contents |
Motivation
In order to apply hydrological models successfully it is necessary to define model parameters accurately. A direct measurement of the parameters is mostly not possible, too expensive or there is no clear physical relation. For those reasons the parameters are adjusted in a trial and error process in so far that the simulated factors (e.g. runoff) correspond best to the values measured. This task can be time-consuming and difficult if the corresponding model is complex or has a large number of parameters.
Requirements
A basic requirement is, of course, a functioning model.
To evaluate the quality of an individual simulation mostly a singular measurement is used. Frequently used examples are the mean square error or the Nash-Sutcliff efficiency. The Jena Adaptable Modelling System the Standard Efficiency Calculator is available for this reason. The image below shows an example configuration of this component
Bild:Eff calc.jpg
The component calculates the efficiencies listed in parameter effMethod with:
- 1 -> Nash - Sutcliff efficiency with power 1
- 2 -> Nash - Sutcliff efficiency with power 2
- 3 -> Nash - Sutcliff efficiency of the logarithmic values with power 1
- 4 -> Nash - Sutcliff efficiency of the logarithmic values with power 2
- 5 -> Index of Agreement (1)
- 6 -> Index of Agreement (2)
- 7 -> regression coefficient
- 8 -> WR2
- 9 -> DSGRAD
- 10 -> absolute volume error
- 11 -> mean square error
- 12 -> PBIAS
- 13 -> PBIAS2
By using the attributes validation and prediction the simulation and validation data are transferred to the component in order to compare them. The time interval in which both time periods should be compared is determined by the attribute effTimeInterval. The modelTimeInterval shows the modeling time. In most applications those two time spans do not correspond. Many models show bad mappings at the beginning of the modeling process since they often need a certain time until internal conditions become stationary. The time of initialising should normally not be considered. Often the qualities of models in a certain time period are of interest (e.g. during snowmelt), so that several efficiency components can be used which consider different time periods.
Calibration
There are two common methods for calibrating a model with the integrated Standard Efficiency Calculator:
Offline Calibration
First, it is possible to carry out a calibration locally on your own computer. However, this uses computing resources for a longer period of time. Step 1
Open the JAMS User Interface Model Editor (JUICE).
Load the model you want to calibrate.
The following window should appear:
Bild:Juice kalibration 1.jpg
Step 2
Select the entry Kalibrierungswizard under the item Modell.
A dialog should appear which offers the configuration of the calibration (see image).
Bild:optimization_wizard.jpg
Step 3
In the middle of the dialog area on the left hand side the potential parameters of the model are listed. Select those parameters from the list which you want to calibrate. If you want to select more than one parameter, hold down CTRL while selecting the parameters. Now you can see several input fields for each parameter on the right hand side. You can choose the lower and upper bound which define the area in which the parameter can vary. In addition, it is possible to define a start value. This makes sense if a parameter is already known which suits as a starting point for the calibration.
Step 4
In the the lower area in the dialog the target function is specified. Either a single criterion or various criteria can be selected (for more than one hold CTRL). By choosing various criteria a multi-criteria optimization problem is created which differs significantly from a (common) one-criterion optimization problem regarding its solution. Since not every optimization method is suitable for a multi-criteria problems either, not all optimizers are at disposal this time.
For every selected optimization criterion a list box appears on the right hand side of this section. It determines whether the criterion will be minimized, maximized, absolutely minimized (i.e. min |f(x)| e.g. for the absolute volume error) or absolutely maximized.
Step 5
When parameters and target criteria are selected, it is necessary to choose an optimization method for the calibration. In the upper area of the dialog you can see a list box which lists several optimization methods. At the moment the following methods are available:
- Shuffled Complex Evolution
- Branch and Bound
- Nelder Mead
- DIRECT
- Gutmann Method
- Random Sampling
- Latin Hypercube Random Sampler
- Gauß Process Optimizer
- NSGA-2
- MOCOM
- Paralleled Random Sampler
- Paralleled SCE
For the application with single target criterion the Shuffled Complex Evolution (SCE) and DIRECT are recommended. Most probably both find a global optimized parameterization. It has been attested that DIRECT shows robust operation and no parameterization is necessary. SCE only needs one parameter: the Anzahl der Komplexe (number of complexes). This parameter indirectly controls whether the parameter search area is browsed rather broadly or if the process quickly concentrates on a (sometimes local) minimum. In most cases, the default value 2 can be used. For multi-criteria optimization NSGA - 2 seems to provide very good results. However, there are more detailed analyses to be carried out.
In addition, there are three check boxes available, independent of the selected method.
- irrelevante Komponenten entfernen: deletes components from the model structure which certainly do not influence the calibration criterion
- GUI Komponenten entfernen: deletes all graphical components from the model structure. This option is strongly recommended since e.g. diagrams have a strong negative influence on the calibration.
- Modellstruktur optimieren: at the moment without function
Step 6
In order to finish the configuration, click on the button XML erzeugen. This closes the dialog and a new (modified) model is set up. You can start the calibration by starting the modeling. Please keep in mind that the calibration assistant modifies your output data stores in such a way that for every model run during calibration model parameters and target criteria are given. If you want to avoid this output, delete the file optimization_wizard_optimizer.xml manually.
Online Calibration
As an alternative to offline calibration the process can be carried out on the computing cluster of the Department of Geoinformatics, Hydrology and Modelling at the Friedrich - Schiller - University Jena. This does not occupy any local computing resources and allows a calibration of up to four models at the same time.
Step 1
Open your Internet browser (e.g. Firefox) and go to website [[1]] You will now see the following window and you will require a login. In order to register please contact
- Christian Fischer
- christian.fischer.2@uni-jena.de
Step 2
After the registration you will see the main window of the application.
Bild:Optas_main.png
The window is split up into three parts. In the middle you can see a list of all tasks that have been started. All tasks which have been finished are marked in blue color and tasks which are being carried out are marked green. Red entries indicate that an error has occurred during the configuration of a new task/calibration. The right part of the window shows different contents. When starting the application, you can see news about the OPTAS project. In general, information up to a size of 1 megabyte is shown. Greater data sizes are not shown because it affects the page view. Instead, a zip file is created from the information which is then available for download.
Step 3
In order to carry out a new calibration, click on "Create Optimization Run". You will now see the first step of the calibration process.
Bild:Optas_step1.png
In this step you will be asked to enter the data which is necessary. Load the model for the calibration (jam file) into the first file dialog. Pack your working directory of the model as a zip archive. A suitable programs is for example 7-zip. Please note:
- the working directory cannot be in a subfolder
- please do not load any unnecessary files (e.g. in the output file). The upload is limited to 20mb.
- the file default.jap is not allowed to be in the working directory
Now load the packed working directory into the second file dialog. The modeling will usually be carried out by using the current version of JAMS/J2000. Libraries for J2000g and J2000-S are usually available as well, so you will not need any files for most applications.
Should you need any versions of component libraries which are particularly adapted for models J2000, J2000-S or J2000g, please load them (unpacked) into the corresponding file dialogs. If your models requires additional libraries, you can indicate them as a zip archive in the corresponding file dialog.
To finish, click on Submit. The files that you indicated are now are now being loaded on the server environment.
The management server (sun) checks the host for free capacities and assigns a host to your Task.
Step 4
After transferring the model and the workspace, the following dialog for the selection of parameters and goal criteria appears.
Bild:Optas_step2.png
The first list shows all potential parameters of the model. Select those parameters which you want to calibrate. If you want to select several parameters, hold CTRL when marking them.
In the low part of the dialog a target function is specified. You can either select a single criterion or several criteria (hold CTRL). When selecting several criteria, a multi-criteria optimization problem is created which differs significantly from a (common) one-criteria optimization problem regarding its solution characteristics. Since not every optimization method is suitable for multi-criteria problems, not all optimizers are available in this case.
Step 5
In this step, the valid range of values is to be specified for every parameter. For every selected parameter an input field is available for the specification for upper and lower boundaries. The fields are occupied by default values, if they are available for the specific parameter. These default values are mostly very broad, so it is recommended to carry out further restrictions. In addition, there is an optional possibility to indicate the start value. This is helpful if an adequate set of parameters is already available which is taken as the starting point of the calibration.
In the second part, you can choose for every criterion whether it should be
- minimized (example RSME - root square mean error),
- maximized (Beispiel r²,E1,E2,logE1,logE2),
- absolutely minimized (i.e. min |f(x)| e.g. for absolute volume error),
- or absolutely maximized.
The following image shows the corresponding dialog. In this example, the parameters of the snow module were chosen and the default thresholds were entered. Bild:Optas_step3.png
Step 6
You are now asked to choose the desired optimization method. At the moment, the following methods are available:
- Shuffled Complex Evolution (Duan et al., 1992)
- Branch and Bound (Horst et al., 2000)
- Nelder Mead
- DIRECT (Finkel, D., 2003)
- Gutmann Method (Gutmann .. ?)
- Random Sampling
- Latin Hypercube Random Sampler
- Gaußprozessoptimierer
- NSGA-2 (Deb et al., ?)
- MOCOM ()
- Paralleled random sampling
- Paralleled SCE
For most one-criterion applications the Shuffled Complex Evolution (SCE) method and the DIRECT method can be recommended. For multi-criteria tasks the NSGA-2 mostly delivers excellent results.
Step 7
Many optimizers can be parametrized according to the specific task. SCE has a freely selectable parameter: the number of complexes. A small value leads to rapid convergence, though with higher risk of not identifying the global solution, whereas a greater value increases calculation time significantly while strongly improving the possibility of finding an optimum. Mostly a default value of 2 can be used to work with. In contrast, DIRECT for example does not possess any parameters. It shows a very robust behavior and no parametrization of the method is required. Moreover, three check boxes are available which are not subject to the specific method.
- remove not used components: removes components from the model structure which definitely do not influence the calibration criterion
- remove GUI-components: removes all graphical components from the model structure. This option is highly recommended since e.g. diagrams influence the efficiency of the calibration in a very negative way
- optimize model structure: at the moment without function
The following image shows step 7 for the configuration of the SCE method.
Step 8
Before starting the optimization you can indicate which variables/attributes of the model should appear in the output. As a default, those parameters and target criteria are shown which have already been chosen which makes manual selection unnecessary. Pleas note that saving temporally and spatially varying attributes (z.B. outRD1 für jede HRU) may require a huge amount and this may exceed the size belonging to a user account. Bild:Optas_step6.png
Step 9
You now receive a summary of all modifications which have been carried out automatically. Particularly, you now have a list of components which are classified as irrelevant. By clicking on Submit you start the optimization process.
Operating OPTAS
After having started the calibration of the model, you should now see a new job on the main page. This is shown green to indicate that the execution is not yet completed. You also see the ID, the start date and the estimated date of completion for the optimization. If you select the generated job, you now have various options at your disposal.
Show XML
You can see the model description for carrying out this job which has automatically been generated. As soon as you click on the button, it appears on the right hand side in the window and can be downloaded from there as well.
Show Infolog
Click on this button in order to view status messages of the current model. When using an offline execution of the model this information is written in the file info.log or error.log.
Show Ini
The optimization-Ini file contains all Datei enthält alle vorgenommen Einstellungen und fasst diese in Kurzschreibweise zusammen. Diese Funktion orientiert sich an dem fortgeschrittenen Benutzer bzw. Entwickler, so dass hier auf eine detailierte Erläuterung verzichtet wird.
Show Workspace
Diese Funktion listet alle Dateien und Verzeichnisse im Workspace auf. Durch einen Klick auf das Verzeichnis/Datei kann dessen Inhalt angezeigt werden. Die Modellergebnisse lassen sich normalerweise im Ordner /current/ finden, den die folgende Abbildung beispielhaft zeigt.
Hierbei ist zu beachten, dass Dateiinhalte nur bis zu einer Maximalgröße von 1 Megabyte dargestellt werden und ansonsten als zip Datei gepackt zum Download zur Verfügung gestellt werden.
Show Graph
Abhängigkeiten zwischen verschiedenen JAMS-Componenten eines Modells können grafisch dargestellt werden. Diese Funktion erzeugt einen gerichteten Graphwn, dessen Knoten die Menge aller Komponenten des Modells sind. In diesem Graphen ist eine Komponente A mit einer Komponente B verknüpft, wenn A von B abhängig ist, in dem Sinne, dass B Daten erzeugt, die direkt von A benötigt werden. Außerdem werden alle Komponenten rot dargestellt, die während der Modellkonfiguration entfernt wurden. Dieser Abhängigkeitsgraph erlaubt die Analyse komplexerer Abhängigkeitsbeziehungen, wie zum Beispiel die Bestimmung von Komponenten, die indirekt abhängig von einem bestimmten Parameter sind. Derartige Analysen sollten jedoch automatisiert durchgeführt werden. Die grafische Ausgabe dieses Graphen dient vorrangig der Veranschaulichung.
Stop Execution Beendet die Ausführung eines oder mehrere Modelläufe.
Delete Workspace Löscht das komplette Arbeitsverzeichnis des Modells auf der Serverumgebung. Bitte verwenden Sie diese Funktion, wenn die Optimierung abgeschlossen ist und sie die Daten gespeichert haben. Die entlastet die Serverumgebung und beschleunigt die Darstellung der OPTAS Umgebung.
Modify & Restart Diese Funktion richtet sich an den fortgeschritten Benutzer. Oft wird ein Neustart eines Modells oder Kalibrierungsaufgabe benötigt, ohne dass große Änderungen der Konfiguration nötig sind. Durch die Funktion Modify & Restart, kann die Konfigurationsdatei der Optimierung direkt verändert werden und anschließend der Neustart des Modells in einem neuen Arbeitsverzeichnis erzwungen werden.
Refresh View Aktualisiert die Ansicht. Bitte Verwenden Sie nicht die Aktualisierungsfunktion ihres Browsers (meist F5), da dies eine erneute Ausführung des letzten Befehls nach sich ziegt (z.B. Start des Modells)
