Troubleshooting calculation performance

When do long calculation times appear

There are a number of moments during the usage of a project where calculations take place. Roughly speaking, whenever a change in the data of a project or session takes place, calculations are performed. The Tygron Platform is already optimized to "cache" many results. If only part of the data changed, only the calculations affected by the changed data are performed again. However, there are situations in which all calculations need to be performed again. You may at some point in your project experience that the Tygron Platform spends a long time performing calculations, despite these mechanisms.

• When loading a project into a single/multi user session*
• When performing an action
• When firing an event
• When switching levels

• When loading a project into the editor*
• When starting or stopping a Test Run*
• When adding/removing/changing anything spatial (zones, buildings, areas, etc)
• When uploading excel sheets

*These are single moments of processing, and should not be considered too harshly.

Probable causes of long calculation times

There are a number of components in a project which, when used in excess, can cause performance issues. Below is a list of the most likely causes, as well as some general advice on how to modify your project to negate the slowdown.

Also note the options in the Calculation panel (see below).

Performance Overview Panel (Dutch only)

Areas (And polygons)

Because many calculations in the Tygron Platform rely on spatial data, many things in the Tygron Platform have some form of polygon: a shape which, in one way or another, is "in" the project area. Buildings, zones and areas are most easily recognized as polygons. But other things such as land ownership and land types do too. When data changes, the Tygron Platform may have to see whether this has affected any polygons. If it has, it means things like the surface area of something has changed, or that some element now is or isn't overlapping with another polygon anymore. The Tygron Platform tries to find ways to avoid these calculations as much as possible, but they must occur at some point. The following situations may cause the Tygron Platform to calculate too much when it comes to polygons.

Many areas (or polygons)

The more polygons exist, the more the Tygron Platform will have to check to see whether they require an update. A (quick) calculation will be performed to see if the change has affected a given polygon, and if it has, a more thorough calculation will be performed to calculate the actual resulting data. If there are more polygons, more will need to be checked that first time, to see whether they are affected by a given change.

Buildings form an exception. They are used less as "containers" for calculations, so aren't checked in the same manner as often.

Where possible, use fewer polygons rather than more. Also, be sure to clean up polygons you no longer use (for example: unused areas). With the exception of buildings, the amount of polygon elements (areas, zones, etc) in a project should not be more than a thousand.

Large areas (or polygons)

The larger a polygon is, the more chance there is of it being affected by a change. Even though the change may be marginal in respect to the size of the polygon, every small change that takes place within it will cause the Tygron Platform to recalculate for that polygon.

Where possible, keep your polygons a reasonable size. As a general rule of thumb, the zones in a project should be the largest polygons you have.

Complex areas (or polygons)

The more complex a polygon is, the more time it takes for the Tygron Platform to calculate whether a change intersects with it or not. Complexity comes from the amount of subpolygons a polygon has (smaller, scattered pieces, or holes within the larger polygon), and the amount of corners a polygon has. A doughnut, 10 meters across, is more complex than a square 200 meters across, because a doughnut's edge is comprised of many small corners, and it has a subpolygon for the hole in the middle.

Where possible, keep polygons as simple as possible. There is no easy measure for determining the complexity of a polygon. If you find a polygon which is long, thin, has many corners, branches or holes, try to get the polygon smoothed out, or cut up into smaller polygons with fewer corners and subpolygons.

Hoe kan het dat mijn project traag is geworden na het inlezen van geodata? (Dutch only)

Excel sheets

The Tygron Platform's calculations are optimized greatly, but with the ability to add your own excel sheets it's possible to add more complex calculations to the Tygron Platform's tasks. If these calculations are not optimized as well, the Tygron Platform will spend more time to perform these calculations. When designing your own calculation sheets, there are a number precautions you can take to prevent the Tygron Platform slowing down too much on them. This can seem tricky, because when opened in Excel a sheet may seem to calculate quite quickly. However, when a different application, such as the Tygron Platform, handles an excel sheet, the calculations will take a bit longer. Under some circumstances, these small increases in calculation time can add up.

For debugging purposes, the Tygron Platform displays the calculation time for excel indicators when they are selected in the editor.

Complex excel formulas

Excel is a powerful tool because of the functions and formulas it provides. However, some of the functions excel provides are more complex than others. For example, functions such as MATCH, which look up an value in a list by comparing it to another value, are more complex than INDEX, which simply returns the value in a given place in a list. It's not an issue to use a computationally heavy function here and there, but where possible they should be used as little as possible. Functions like IF and SUMIF should be avoided as well where possible, though they are not as problematic.

One concrete example is that often MATCH and INDEX are used together to look up a value and return something on the same row or column. If multiple values are to be retrieved from that same row, rather than using MATCH multiple times to determine the row number, use it once and store the row number in a separate cell. Then every time you need to retrieve values from that row, refer to that cell.

In general, avoid comparative lookups in excel sheets. One effective rule of thumb is to make your calculations as linear as possible. I.e. For a calculation of any given area, place all retrieved data and the calculations on a single row, moving from left to right as the calculation progresses.

Calltree recording

To help debugging excel files to find significant calculation times, you can use the Calltree recording function. During a Test Run in the editor, this option will cause the Tygron Platform to track how long it takes to calculate each cell in the sheet.

X-queries

TQL offers the ability to create X-queries; queries in which the referred ID for a single item is replaced by an X, indicating that you wish to apply that query for all valid IDs. When an excel sheet is uploaded with an X-query, the Tygron Platform prepares the excel sheet by transforming the single X-query into all the implied queries. This means that a single query can lead to hundreds of queries, if there are hundreds of valid IDs. Depending on how many X-queries are added to a single excel sheet, it's possible to quickly cause the sheet to perform thousands of queries.

When using X-queries, make sure to use as few as possible. Do not request data which you do not use. Additionally, when you find that you really only need a few of the many queries, and you can predict or control which ones they are, it can help greatly to replace the X-queries with regular queries.

Overlays

Overlays come in a variety of forms. Naively, they can be seen as 2D images which display information. But depending on the exact overlay in question, a lot more can be going on. There are overlays which indeed just show a relatively static image. There are also a number which rely on coloring, or not coloring, polygon data in the project. These take some minor calculations, but don't often make any significant impact. The most significant slowdown occurs when the Tygron Platform has to perform a lot of processing to turn the current project data into a visual layer.

Grid overlays

A few Overlays in the Tygron Platform are actually calculation models. These include, among others, the overlays for livability, subsidence, heat, and traffic noise. These calculations happen in the form of a grid; parts of which are updated whenever their underlying data is changed. The size of the cells of those grids can be changed. The smaller the cells, the more accurate the calculations and the shapes generated by the overlay. However, smaller cells mean more calculations. Selecting a smaller cell size means the Tygron Platform will spend a greater amount of time to recalculate (parts of) an overlay.

Where possible, set the grid size of calculated as low as is required for your use-case but no lower.

Heavy calculations

Some calculation models, such as the flooding overlay, have an inherently more complicated calculation. Especially iterative processes, in which calculations are repeated a large number of times. In combination with an accurate grid size, these calculations can take a long time to complete. The more data the calculation relies on, the longer the system will be occupied preparing the data for the calculation. The more interactions each grid cell has to calculate, and the more iterations need to be calculated, the more calculations will need to be made in total. And the more output the overlay is set to create, the longer it will take to transfer all new data back to the end-user.

Note that these overlays may have more calculation- and implementation- specific settings which balance speed and accuracy. Tweaking these settings will affect the outcome of the calculations, but represent a trade-off between responsiveness and detail.

Where possible, avoid using these overlays in sessions where real-time responsiveness (i.e. seconds) is vital. For planery or planning session, it is attainable to configure these overlays to calculate in a few minutes, if concessions to accuracy are made.

Multiple results

Some overlays, such as the subsidence overlay or the flooding overlay, perform calculations which may generate a variety of relevant results. For each instance of such an overlay, a complete calculation must be performed to get the results for each. This is because the settings for each overlay can differ. Even if the only differing aspect is the result or output type, they are still considered separate calculations. Overlays with multiple result types offer a configuration where multiple result types can be added to such an overlay as child overlays. These are not entirely separate overlays, but separate output layers for the same calculation.

Where possible, avoid creating separate calculating overlays with (virtually) unchanged parameters. Where possible, use the child result types to provide multiple types of output, and take care to make sure that they are child results of a single main calculation.