DEPARTMENT OF INFRASTRUCTURE ENGINEERING UNIVERSITY OF MELBOURNE ENEN90031Quantitative Environmental Modelling
Modelling Assignment 2
Due Midnight on Friday 5th June 2020.
Introduction
This assignment concentrates on the use of global model analysis techniques that can be applied to optimisation, sensitivity analysis and model uncertainty analysis. The final step involves applying your model results to determine a cost-reliability relationship for a hydroelectric power station at the catchment outlet and to estimate the impact of climate change on that relationship. The key parts of the assignment are:
1 Global optimisation of your model using Shuffled Complex Evolution;
2 Undertaking a regional sensitivity analysis to understand the effects of different parameters on the model performance;
3 Undertaking an uncertainty analysis using GLUE; and
4 Assessing the cost-reliability relationship of a potential hydroelectric power plant and the impact of climate change on that relationship.
The assessment for this assignment is as follows.
• Model calibration (section 1): (Technical 10%, Discussion 10%)
• Regional sensitivity analysis (section 2) (Technical 10%, Discussion 10%)
• Parameter uncertainty analysis (section 3) (Technical 10%, Discussion 10%)
• Power plant assessment (section 4) (Technical 10%, Discussion 10%)
• Discussion and conclusions (Discussion 10%)
• Quality of report presentation 10%
Your report should contain the following: aims, introduction, discussion, conclusions, references and an appendix containing your MATLAB code. The report must be no more than 12 pages (including graphs but excluding the appendix) and any additional pages will not be assessed. In writing your report, try to demonstrate to the reader your knowledge of the methodologies and their limitation using academic references and modelling results. Please submit your report using the LMS assignment link.
Engineering Practice Hurdle
This assignment can be used as the final piece of your Engineering Practice Hurdle Written
1 /6
Communication submission. STEP workshops and online lessons are available to help you further develop your writing skills.
See the Skills Towards Employment Program community for more details on the Engineering Practice Hurdle.
Code structure and examples
There are two places you can get information on how all the different functions fit together and how to run them: the headers in each function and in the document “Overview of functions used in Modelling assignment 2”. Please note there are differences compared to past years.
Time Periods
Please use 2005-2018 for parts 1-3 of the assignment. Please use 1988-2018 for part 4 of the assignment.
1 Global Calibration using Shuffled Complex Evolution (SCE)
Please use a daily timestep for objective function evaluation (doMonthly = false)
Using the SCE algorithm, the model parameters can be adjusted to best reproduce the observed data. For all but the most trivial of models finding the global optima is very challenging. The constants for the SCE algorithm can significantly affect the results and the parameter range in which the algorithm searches may not include the global optima. For this section, please apply the SCE algorithm (code is supplied) and assess whether you have achieved a reliable.
This can be done as follows.
1. As good practice you should show that the results of any optimisation are independent of the settings for the optimisation algorithm. We have given you reliable values for kstop, pcento and maxn; however, you need to show that your solution is independent of the number of complexes
i.e. SCE setting ngs. You should demonstrate that SCE is reaching a similar optima when different values of ngs are used. To do this run SCE with a variety of values of ngs (say 10) and plot the resulting optimal objective function value. The algorithm should reach a consistent result as ngs increases.
2. With optimisation you should always evaluate the repeatability of you results for given settings. To do this you should run SCE several times for you final ngs and show that you get practically consistent (not necessarily identical, given there is some randomness in the algorithm) results when you rerun SCE using identical settings.
For this section, your report should:
3. outline your calibration procedure;
4. provide evidence to support your estimate of the global optima (i.e. why do you think you have found the global optima?);
5. comment on the adequacy and limitations of the calibration scheme;
6. provide an evaluation of the model and in particular estimate the normalized sum of squared 2
/6
error = SSE / sum[(Observed – mean(Observed))]2 (noting that objectiveFunction.m outputs SSE); and
7. critique your overall approach.
2 Regional sensitivity analysis (RSA)
The aim of Regional Sensitivity Analysis is to understand the relationships between the model predictions and the parameters using a form of global sensitivity analysis. This is based on a Monte Carlo analysis of the model. We have supplied Regional Sensitivity Analysis code that you can use. This code outputs the parameters and performance assessments for each Monte Carlo run. You should generate a set of runs and save the results of those runs for later analysis. There is always a trade-off between run time and the number of simulations conducted (and hence the precision of your later analyses). We are not expecting you to use more than one hour of computation time to produce your final set of Monte Carlo model runs.
See the code overview for some hints on reusing model runs to save time.
It is a good idea to save the following to allow you to reuse model runs.
• The parameter values for each run;
• The simulated output (flow) and
• The model performance metrics you need to analyse.
For this section, your report should provide an interpretation of results of the RSA (both the cumulative distribution function plots and the plotmatrix plots) in terms of:
3
• how sensitive the model results are to each parameter;
• whether there are significant parameter interactions;
• which parameters are most likely to be identifiable through optimisation;
• what limitations there might be for this analysis; and
• what implications there are for choosing GLUE threshold levels
Parameter uncertainty analysis
Having now undertaken both the global optimisation and the sensitivity analysis, it is now time to focus in on how uncertain each of the parameters you have identified is and what implications this has for uncertainty in the model predictions. This involves further using your set of Monte Carlo runs to undertake a parameter uncertainty analysis and rerunning the model for a subset of the Monte Carlo runs to explore how the parameter uncertainty feeds through to prediction uncertainty. To estimate the parameter uncertainty and prediction uncertainty, please use the Generalised Likelihood Uncertainty Estimation (GLUE) methodology.
An important aspect of GLUE is the selection of the threshold performance to consider a run as behavioural. In the light of your optimisation and the exploration of the model in the RSA, you need to choose a threshold performance level. This requires some judgement about what is an acceptable and what is not an acceptable model performance and this decision should be discussed when you write up your report. Those model runs that meet the threshold (perform well enough) are then identified and their parameters obtained. You can gain some insight into an appropriate level for the
3 /6
threshold by comparing the uncertainty limits with the observed data; however, the GLUE analysis does not necessarily represent all uncertainty. Lastly, an estimate of the uncertainty is achieved by summarising the parameters that result in an acceptable simulation.
To help you apply GLUE, we have written MATLAB code for the GLUE framework. See the MatLab code overview document for more details. You should analyse the estimated parameter uncertainties and critique your overall approach. Below are some points to consider when running and/or critiquing your results.
1. Using presaved rainfall-runoff model runs will be much quicker if you are trialing different options for thresholds etc – see MatLab code overview document. Be aware that if you change the parameter bounds you would need to rerun the rainfall runoff model.
2. We have supplied upper and lower bounds for each parameter. You should assess whether you think these are satisfactory, too narrow or too wide and whether your results are independent of this decision. You could consider the results of your RSA as a preliminary basis for making this decision and then revisit it following your GLUE analysis.
3. The number of parameter samples (realisations) determines how well the response surface is characterised. Set this to large enough that your solutions are independent of this setting but no so large that GLUE takes a prohibitively long time to run. There may be trade-offs in setting the number of samples and the reliability that should be discussed in your report.
4. Implementing GLUE requires a choice of the maximum acceptable error for a model to be considered behavioural, which can be challenging. Please clearly discuss your choice.
5. To summarise the parameter uncertainties there are a number of options. One option is to create box-plots for each parameter that communicates the parameter distributions. Another option is to plot the cumulative likelihood against parameter values. As such plots do not communicate the parameter interactions, you may also want to consider a more advanced plotting function called ‘plotmatrix’. Plotmatrix produces a histogram for each parameter as well as scatter plots of each parameter against the others to communicate parameter interactions (see the RSA code for an example).
Note the similarities in RSA and GLUE and take this into account when writing your report. For example there is no need to repeat figures that have very similar content and your discussion should not repeat the same/similar things.
Once you have completed the above, discuss the findings, any parameter interactions, and the adequacy of your method. You may want to review Beven (2001) to better understand the GLUE methodology.
4 Hydroelectric power plant cost-reliability trade-off and climate change assessment
Note for this section reliability should be characterized by using demandReliability output of hydropower. This is also the output from hydroDesign.
Use 1988-2018 for part 4.
Now that the model has been calibrated, the model can be used to make predictions. By also using 4
/6
the parameter uncertainties, the prediction error can be quantified. This part of the assignment looks at the cost and reliability of a hydroelectric power plant used to meet peak demands. The demandData variable supplied has the component of the overall electricity demand that the hydroelectric generation needs to meet. The hydroDesign function optimizes the allocation of a capital budget between water storage infrastructure and other infrastructure to maximise reliability. Obviously the more you spend, the more reliable a system might be, a least up to a point. Here we first want to determine the cost-reliability trade-off.
Such a facility would have a long life so it is also important to estimate future reliability for each system. The rainfall runoff model can be used to estimate changes in flows with changed climate and the hydropower function can then be used to see how the performance of the hydroelectric system might change. The hydroPower function calculates both the reliability in time (the proportion of days demand is fully met) and also the proportion of the peak demand that the scheme is able to meet. Both the impacts of model uncertainty and climate change lead to uncertainty in the hydroelectric system operation.
There are two tasks here:
1. Using the model to develop a cost-reliability curve for a future hydroelectric system. By running hydroDesign multiple times with different budgets, we can develop such a curve. By using the behavioural models and simulating for the longer period (1988-2018), we can also get an estimate of the uncertainty in reliability estimates for each budget (one reliability for each behavioural parameter set is generated).
Catchment area = 361 km2 Head = 300m
2. Assess the impact of climate change on the reliability of the hydropower generation by calculating the reliability of the design in meeting the energy demands under a 15% reduction in rainfall and 5% increase in PET.
In writing up your assessment of the hydroelectric power system reliability, you should evaluate how well you think these uncertainties have been captured and comment on potential improvements in the method that you can think of. Comment on other potential sources of uncertainty.
5 Discussion and conclusions
Lastly, in your report you need to discuss the strengths and weakness of your modelling for this assignment and summarise the findings from each part of the model analysis including the reliability of your predictions.
6 References
Beven, K. (2001), Rainfall-runoff modelling: the primer, J. Wiley, , xi, 360
5 /6