Troubleshooting calculation performance and Groundwater Overlay: Difference between pages

From Tygron Support wiki
(Difference between pages)
Jump to navigation Jump to search
mNo edit summary
 
 
Line 1: Line 1:
{{learned|when performance issues may occur|how to resolve performance issues}}
{{stub}}
==When do long calculation times appear==
There are a number of moments during the usage of a [[project]] where calculations take place. Roughly speaking, whenever a change in the data of a project or session takes place, calculations are performed. The {{software}} is already optimized to "cache" many results. If only part of the data changed, only the calculations affected by the changed data are performed again. However, there are situations in which all calculations need to be performed again. You may at some point in your project experience that the {{software}} spends a long time performing calculations, despite these mechanisms.


* When loading a project into a single/multi user session*
The groundwater overlay is an implementation of the [[Water model]] optimized for calculating the flow of underground watertables. This implementation focuses on water flowing at a low speed, and on the functioning of the water system without calamitous events such as [[Water model#Breach|breaches]].
* When performing an [[action]]
* When firing an [[event]]
* When switching [[levels]]


* When loading a project into the [[editor]]*
==Use cases==
* When starting or stopping a [[Test Run|test run]]*
The groundwater overlay can be used and is recommended when exploring the following use-cases:
* When adding/removing/changing anything spatial ([[zones]], [[constructions]], [[areas]], etc)
* Modeling a static, low-speed water system
* When uploading excel sheets
* Modeling the long-term effects of excessive rain or excessive drought on the water table
* Modeling the spread of chemicals through the underground, carried along by water


''*These are single moments of processing, and should not be considered too harshly.''
==Preparing to use the overlay==
To use the overlay, it is recommended that the project meets a number of prerequisites. Next, data can be prepared which define the functioning and flow of the water model. This includes a clear idea of what kind out output is desired. After that data is prepared, you can begin the actual steps of adding the overlay.


==Probable causes of long calculation times==
It is important to model a fully functional static water system for any relevant use of the groundwater overlay. After setting up the static water model, additional effects can be modeled to account for special circumstances. This has the added benefit of creating a control-state, to which results of modifications or calamities can be compared.
There are a number of components in a project which, when used in excess, can cause performance issues. Below is a list of the most likely causes, as well as some general advice on how to modify your project to negate the slowdown.


===Prerequisites===
When creating your [[project]], make sure it meets the following [[Water model#Usage|criteria]]:
* Your project has been loaded in with a high-resolution DEM. This can be configured during the [[new project wizard]].
* Your project is large enough to account for edge effects. A buffer of at least 1km around your actual project area is recommended.
* The project's eventual use. The calculations will take longer than most other calculations in the {{software}}, so sessions with live interaction are discouraged.


===Data===
Before creating and configuring the overlay, it is best to prepare the data you wish to use in your model.


===Areas (And polygons)===
====Geographical data====
Because many calculation in the {{software}} rely on spatial data, many things in the {{software}} have some form of polygon: a shape which, in one way or another, is "in" the [[3D environment]]. Constructions, zones and areas are most easily recognized as polygons. But other things such as land [[ownership]] and [[terrain|land types]] do too. When data changes, the {{software}} may have to see whether this has affected any polygons. If it has, it means things like the surface area of something has changed, or that some element now is or isn't overlapping with another polygon anymore. The {{software}} tries to find ways to avoid these calculations as much as possible, but they must occur at some point. The following situations may cause the {{software}} to calculate too much when it comes to polygons.
The following data needs to be prepared, so that it can be loaded in and used by the water model's calculations:
{{OverlayChecklist|table=true}}
{{OverlayChecklist|Water areas|link=Water model#Water_area|
  importance=Required for proper water levels|
  description=Water areas set the water level of all open water in the area they envelop.}}
{{OverlayChecklist|Ground water|link=Water model#Groundwater_geotiff|
  importance=Required for proper water levels|
  description=The ground water geotiff sets the underground water level.}}
{{OverlayChecklist|Weirs|link=Water model#Weir|
  importance=Required for proper water flow|
  description=Weirs connect otherwise separate sections of open water, allowing water to flow over the weir.}}
{{OverlayChecklist|Culverts|link=Water model#Culvert|
  importance=Required for proper water flow|
  description=Culverts connect otherwise separate sections of open water, allowing water to flow through the culvert.}}
{{OverlayChecklist|Pumps|link=Water model#Pump|
  importance=Required for proper water flow|
  description=Pumps connect otherwise separate sections of open water, moving water unidirectionally from the lower terrain level to the higher terrain level.}}
{{OverlayChecklist|Inlets|link=Water model#Inlet|
  importance=Required for proper water flow|
  description=Inlets are points in the water model where water is added or removed from the hydrological system.}}
{{OverlayChecklist|Sewer areas|link=Water model#Sewer|
  importance=Recommended for proper water levels|
  description=Sewer areas intersect with [[sewered]] [[constructions]] in the area they envelop to form sewers which can store water.}}
{{OverlayChecklist|Sewer overflows|link=Water model#Sewer_overflow|
  importance=Recommended for proper water flow|
  description=Sewer overflows allow water to flow from a sewer out into open water.}}
{{OverlayChecklist|Chemical producers|link=Water model#Chemical_hotspots|
  importance=Required for modelling chemical flows|
  description=Chemical producers create some quantity of chloride, phosphorus, or nitrogen.}}
{{OverlayChecklist|table-end=true}}


=====Many areas (or polygons)=====
====Calculation properties====
The more polygons exist, the more the {{software}} will have to check to see whether they require an update. A (quick) calculation will be performed to see if the change has affected a given polygon, and if it has, a more thorough calculation will be performed to calculate the actual resulting data. If there are more polygons, more will need to be checked that first time, to see whether they are affected by a given change.
Consider the duration you want your simulation to last, and whether you wish to account for rainfall in your model. The following environmental data needs to be prepared, so that it can be configured and used by the water model's calculations:
{{OverlayChecklist|table=true}}
{{OverlayChecklist|Simulation time|link=Water model#Simulation_time|
  importance=Required|
  description=How long should the simulation last in total. Groundwater calculations often look at timespans of weeks or months.}}
{{OverlayChecklist|Timeframes|link=Water model#Timeframes|
  importance=Recommended|
  description=How many result snapshots should be generated by the calculation, for inspection after the calculation completes.}}
{{OverlayChecklist|Rain|link=Water model#Weather|
  importance=Optional|
  description=How long should rain last at the start of the simulation, and how much rain should fall in total.}}
{{OverlayChecklist|table-end=true}}


Constructions form an exception. They are used less as "containers" for calculations, so aren't checked in the same manner as often.
====Result types====
After the calculation completes, consider what kind of output is relevant to your use-case. The following [[Water model#Result_types|result types]] can be considered, so that it can be configured and used by the water model's calculations:
{{OverlayChecklist|table=true}}
{{OverlayChecklist|SURFACE_LAST_VALUE|link=Water model#Result_types|
  importance=Required|
  description=The amount of water in any given location.}}
{{OverlayChecklist|UNDERGROUND_LAST_VALUE|link=Water model#Result_types|
  importance=Required|
  description=The groundwater level rise in any given location.}}
{{OverlayChecklist|SURFACE_FLOW|link=Water model#Result_types|
  importance=Recommended|
  description=The flow of water across the land and water in total.}}
{{OverlayChecklist|EVAPORATED|link=Water model#Result_types|
  importance=Recommended|
  description=The amount of water which evaporated in total.}}
{{OverlayChecklist|UNDERGROUND_WATERTABLE|link=Water model#Result_types|
  importance=Recommended|
  description=The groundwater level in any given location.}}
{{OverlayChecklist|CHLORIDE, NITROGEN, PHOSPHORUS|link=Water model#Result_types|
  importance=Optional|
  description=The amount of certain compounds present in any location.}}
{{OverlayChecklist|table-end=true}}


Where possible, use fewer polygons rather than more. Also, be sure to clean up polygons you no longer use (for example: unused areas). With the exception of constructions, the amount of polygon elements (areas, zones, etc) in a project should not be more than a thousand.
==Adding and configuring the overlay==
{{Editor ribbon|header=current situation|bar=Overlays|dropdown=Groundwater overlay}}


=====Large areas (or polygons)=====
{{Editor steps|title=use the groundwater overlay
The larger a polygon is, the more chance there is of it being affected by a change. Even though the change may be marginal in respect to the size of the polygon, every small change that takes place within it will cause the {{software}} to recalculate for that polygon.
|Check that your project sufficiently matches the stated {{inlink|prerequisites}}
|Make sure that you have prepared all {{inlink|data}} you wish to use
|Add the groundwater overlay to your project
|Start and follow the steps in the [[Water model#Configuration_wizard|configuration wizard]]
|When the configuration wizard is completed, the overlay will automatically recalculate. (If it does not, select [[Grid_overlay#Grid_recalculation|refresh grid]].)
|When the recalculation has completed, you can inspect the results
}}


Where possible, keep your polygons a reasonable size. As a general rule of thumb, the zones in a project should be the largest polygons you have.
==Data analysis==
When the calculation has completed, results of the calculation inspected by selecting the overlay in the [[session interface]].


=====Complex areas (or polygons)=====
===Overlay map===
The more complex a polygon is, the more time it takes for the {{software}} to calculate whether a change intersects with it or not. Complexity comes from the amount of subpolygons a polygon has (smaller, scattered pieces, or holes within the larger polygon), and the amount of corners a polygon has. A doughnut, 10 meters across, is more complex than a square 200 meters across, because a doughnut's edge is comprised of many small corners, and it has a subpolygon for the hole in the middle.
To check directly whether the model functions as expected, you can activate the overlay, and see the end-results directly. If multiple [[Water_model#Timeframes|timeframes]] have been recorded, the results can be replayed, to see the progression of the results over time.


Where possible, keep polygons as simple as possible. There is no easy measure for determining the complexity of a polygon. If you find a polygon which is long, thin, has many corners, branches or holes, try to get the polygon smoothed out, or cut up into smaller polygons with fewer corners and subpolygons.
===Water balance===
Besides directly inspecting the results geographically, when verifying the functioning of a hydrological model it can be insightful to check the [[Water_model#Water balance|water balance]]. This option allows a net tally of where water ends up in the hydrological system.


==Refining the model==
Based on the results from the initial calculation, you can tweak your data to refine your model. You can update your relevant datasets, and then step through the configuration wizard again.


{{Editor steps|title=refine the groundwater overlay
|Reopen the configuration wizard of the overlay
|In any step where you want to update your data, either change the individual values or clear, and reupload the data
|Skip through all steps which concern data you do not wish to change
|When you finish the configuration wizard, the overlay will automatically recalculate. (If it does not, select [[Grid_overlay#Grid_recalculation|refresh grid]].)
|When the recalculation has completed, you can inspect the results
}}


===Excel sheets===
Every step where no data has changed can be skipped, and the dataset which has been uploaded can be reuploaded in the relevant step. Alternatively, you can do this directly by changing the relevant data in your project. For example, it is possible to directly change the [[attributes]] of [[constructions]].
The {{software}}'s calculations are optimized greatly, but with the ability to add your own excel sheets it's possible to add more complex calculations to the {{software}}'s tasks. If these calculations are not optimized as well, the {{software}} will spend more time to perform these calculations. When designing your own calculation sheets, there are a number precautions you can take to prevent the {{software}} slowing down too much on them. This can seem tricky, because when opened in Excel a sheet may seem to calculate quite quickly. However, when a different application, such as the {{software}}, handles an excel sheet, the calculations will take a bit longer. Under some circumstances, these small increases in calculation time can add up.


For debugging purposes, the {{software}} displays the calculation time for excel indicators when they are selected in the editor.
===Saving the results===
 
If the results of the calculation provide a proper baseline hydrological model, the result of the overlay can be [[Grid_overlay#Save_overlay_result|saved]] as an inactive copy. This means that the overlay, including its direct settings and its results, are duplicated into a new overlay. The duplicate is then set to "inactive". This means the data in that overlay won't change as further changes are made. This means you can set up a new scenario in your original overlay, and after performing another calculation compare the results of the first and the second situation.
=====Complex excel formulas=====
Excel is a powerful tool because of the functions and formulas it provides. However, some of the functions excel provides are more complex than others. For example, functions such as MATCH, which look up an value in a list by comparing it to another value, are more complex than INDEX, which simply returns the value in a given place in a list. It's not an issue to use a computationally heavy function here and there, but where possible they should be used as little as possible. Functions like IF and SUMIF should be avoided as well where possible, though they are not as problematic.
 
One concrete example is that often MATCH and INDEX are used together to look up a value and return something on the same row or column. If multiple values are to be retrieved from that same row, rather than using MATCH multiple times to determine the row number, use it once and store the row number in a separate cell. Then every time you need to retrieve values from that row, refer to that cell.
 
In general, avoid comparative lookups in excel sheets. One effective rule of thumb is to make your calculations as linear as possible. I.e. For a calculation of any given area, place all retrieved data and the calculations on a single row, moving from left to right as the calculation progresses.
 
======Calltree recording======
To help debugging excel files to find significant calculation times, you can use the [[Calltree recording]] function. During a [[testrun]] in the [[editor]], this option will cause the {{software}} to track how long it takes to calculate each cell in the sheet.
 
=====X-queries=====
[[TQL]] offers the ability to create X-queries; queries in which the referred ID for a single item is replaced by an X, indicating that you wish to apply that query for all valid IDs. When an excel sheet is uploaded with an X-query, the {{software}} prepares the excel sheet by transforming the single X-query into all the implied queries. This means that a single query can lead to hundreds of queries, if there are hundreds of valid IDs. Depending on how many X-queries are added to a single excel sheet, it's possible to quickly cause the sheet to perform thousands of queries.
 
When using X-queries, make sure to use as few as possible. Do not request data which you do not use. Additionally, when you find that you really only need a few of the many queries, and you can predict or control which ones they are, it can help greatly to replace the X-queries with regular queries.
 
 
 
===Overlays===
[[Overlays]] come in a variety of forms. Naively, they can be seen as 2D images which display information. But depending on the exact overlay in question, a lot more can be going on. There are overlays which indeed just show a relatively static image. There are also a number which rely on coloring, or not coloring, polygon data in the project. These take some minor calculations, but don't often make any significant impact. The most significant slowdown occurs when the {{software}} has to perform a lot of processing to turn the current project data into a visual layer.
 
=====Grid overlays=====
A few [[overlays]] in the {{software}} are actually [[Grid overlay|calculation models]]. These include, among others, the overlays for [[Livability (Overlay)|livability]], [[Subsidence (Overlay)|subsidence]], [[Heat (Overlay)|heat]], and [[Traffic noise (Overlay)|traffic noise]]. These calculations happen in the form of a grid; parts of which are updated whenever their underlying data is changed. The size of the cells of those grids can be changed. The smaller the cells, the more accurate the calculations and the shapes generated by the overlay. However, smaller cells mean more calculations. Selecting a smaller cell size means the {{software}} will spend a greater amount of time to recalculate (parts of) an overlay.
 
Where possible, set the grid size of calculated as low as is required for your use-case but no lower.
 
=====Heavy calculations=====
Some calculation models, such as the flooding overlay, have an inherently more complicated calculation. Especially iterative processes, in which calculations are repeated a large number of times. In combination with an accurate grid size, these calculations can take a long time to complete. The more data the calculation relies on, the longer the system will be occupied preparing the data for the calculation. The more interactions each grid cell has to calculate, and the more iterations need to be calculated, the more calculations will need to be made in total. And the more output the overlay is set to create, the longer it will take to transfer all new data back to the end-user.
 
Note that these overlays may have more calculation- and implementation- specific settings which balance speed and accuracy. Tweaking these settings will affect the outcome of the calculations, but represent a trade-off between responsiveness and detail.
 
Where possible, avoid using these overlays in sessions where real-time responsiveness (i.e. seconds) is vital. For planery or planning session, it is attainable to configure these overlays to calculate in a few minutes, if concessions to accuracy are made.
 
=====Multiple results=====
Some overlays, such as the [[Subsidence (Overlay)|subsidence overlay]] or the [[Flooding (Overlay)|flooding overlay]], perform calculations which may generate a variety of relevant results. For each instance of such an overlay, a complete calculation must be performed to get the results for each. This is because the settings for each overlay can differ. Even if the only differing aspect is the result or output type, they are still considered separate calculations. Overlays with multiple result types offer a configuration where multiple result types can be added to such an overlay as child overlays. These are not entirely separate overlays, but separate output layers for the same calculation.
 
Where possible, avoid creating separate calculating overlays with (virtually) unchanged parameters. Where possible, use the child result types to provide multiple types of output, and take care to make sure that they are child results of a single main calculation.

Revision as of 08:07, 4 January 2019

This article is a stub.

The groundwater overlay is an implementation of the Water model optimized for calculating the flow of underground watertables. This implementation focuses on water flowing at a low speed, and on the functioning of the water system without calamitous events such as breaches.

Use cases

The groundwater overlay can be used and is recommended when exploring the following use-cases:

  • Modeling a static, low-speed water system
  • Modeling the long-term effects of excessive rain or excessive drought on the water table
  • Modeling the spread of chemicals through the underground, carried along by water

Preparing to use the overlay

To use the overlay, it is recommended that the project meets a number of prerequisites. Next, data can be prepared which define the functioning and flow of the water model. This includes a clear idea of what kind out output is desired. After that data is prepared, you can begin the actual steps of adding the overlay.

It is important to model a fully functional static water system for any relevant use of the groundwater overlay. After setting up the static water model, additional effects can be modeled to account for special circumstances. This has the added benefit of creating a control-state, to which results of modifications or calamities can be compared.

Prerequisites

When creating your project, make sure it meets the following criteria:

  • Your project has been loaded in with a high-resolution DEM. This can be configured during the new project wizard.
  • Your project is large enough to account for edge effects. A buffer of at least 1km around your actual project area is recommended.
  • The project's eventual use. The calculations will take longer than most other calculations in the Tygron Platform, so sessions with live interaction are discouraged.

Data

Before creating and configuring the overlay, it is best to prepare the data you wish to use in your model.

Geographical data

The following data needs to be prepared, so that it can be loaded in and used by the water model's calculations: Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist

Calculation properties

Consider the duration you want your simulation to last, and whether you wish to account for rainfall in your model. The following environmental data needs to be prepared, so that it can be configured and used by the water model's calculations: Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist

Result types

After the calculation completes, consider what kind of output is relevant to your use-case. The following result types can be considered, so that it can be configured and used by the water model's calculations: Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist Template:OverlayChecklist

Adding and configuring the overlay

Template:Editor ribbon

How to use the groundwater overlay:
  1. Check that your project sufficiently matches the stated prerequisites
  2. Make sure that you have prepared all data you wish to use
  3. Add the groundwater overlay to your project
  4. Start and follow the steps in the configuration wizard
  5. When the configuration wizard is completed, the overlay will automatically recalculate. (If it does not, select refresh grid.)
  6. When the recalculation has completed, you can inspect the results

Data analysis

When the calculation has completed, results of the calculation inspected by selecting the overlay in the session interface.

Overlay map

To check directly whether the model functions as expected, you can activate the overlay, and see the end-results directly. If multiple timeframes have been recorded, the results can be replayed, to see the progression of the results over time.

Water balance

Besides directly inspecting the results geographically, when verifying the functioning of a hydrological model it can be insightful to check the water balance. This option allows a net tally of where water ends up in the hydrological system.

Refining the model

Based on the results from the initial calculation, you can tweak your data to refine your model. You can update your relevant datasets, and then step through the configuration wizard again.

How to refine the groundwater overlay:
  1. Reopen the configuration wizard of the overlay
  2. In any step where you want to update your data, either change the individual values or clear, and reupload the data
  3. Skip through all steps which concern data you do not wish to change
  4. When you finish the configuration wizard, the overlay will automatically recalculate. (If it does not, select refresh grid.)
  5. When the recalculation has completed, you can inspect the results

Every step where no data has changed can be skipped, and the dataset which has been uploaded can be reuploaded in the relevant step. Alternatively, you can do this directly by changing the relevant data in your project. For example, it is possible to directly change the attributes of constructions.

Saving the results

If the results of the calculation provide a proper baseline hydrological model, the result of the overlay can be saved as an inactive copy. This means that the overlay, including its direct settings and its results, are duplicated into a new overlay. The duplicate is then set to "inactive". This means the data in that overlay won't change as further changes are made. This means you can set up a new scenario in your original overlay, and after performing another calculation compare the results of the first and the second situation.