Troubleshooting calculation performance

When do long calculation times appear

There are a number of moments during the usage of a project where calculations take place. Roughly speaking, whenever a change in the data of a project or session takes place, calculations are performed. The engine is already optimized to "cache" many results. If only part of the data changed, only the calculations affected by the changed data are performed again. However, there are situations in which all calculations need to be performed again. You may at some point in your project experience that the engine spends a long time performing calculations, despite these mechanisms.

• When loading a project into a single/multi user session
• When loading a project into the editor
• When starting or stopping a test run
• When performing an action
• When adding/removing/changing anything spatial (zones, constructions, areas, etc)
• When uploading excel sheets
• When switching levels
• When firing an event

Probable causes of long calculation times

There are a number of components in a project which, when used in excess, can cause performance issues. Below is a list of the most likely causes, as well as some general advice on how to modify your project to negate the slowdown.

Areas (And polygons)

Because many calculation in the engine rely on spatial data, many things in the engine have some form of polygon: a shape which, in one way or another, is "in" the 3D environment. Constructions, zones and areas are most easily recognized as polygons. But other things such as land ownership and terrain types do too. When data changes, the engine may have to see whether this has affected any polygons. If it has, it means things like the surface area of something has changed, or that some element now is or isn't overlapping with another polygon anymore. The engine tries to find ways to avoid these calculations as much as possible, but they must occur at some point. The following situations may cause the engine to calculate too much when it comes to polygons.

Many areas (or polygons)

The more polygons exist, the more the engine will have to check to see whether they require an update. A (quick) calculation will be performed to see if the change has affected a given polygon, and if it has, a more thorough calculation will be performed to calculate the actual resulting data. If there are more polygons, more will need to be checked that first time, to see whether they are affected by a given change.

Constructions form an exception. They are used less as "containers" for calculations, so aren't checked in the same manner as often.

Where possible, use fewer polygons rather than more. Also, be sure to clean up polygons you no longer use (for example: unused areas). With the exception of constructions, the amount of polygon elements (areas, zones, etc) in a project should not be more than a thousand.

Large areas (or polygons)

The larger a polygon is, the more chance there is of it being affected by a change. Even though the change may be marginal in respect to the size of the polygon, every small change that takes place within it will cause the engine to recalculate for that polygon.

Where possible, keep your polygons a reasonable size. As a general rule of thumb, the zones in a project should be the largest polygons you have.

Complex areas (or polygons)

The more complex a polygon is, the more time it takes for the engine to calculate whether a change intersects with it or not. Complexity comes from the amount of subpolygons a polygon has (smaller, scattered pieces, or holes within the larger polygon), and the amount of corners a polygon has. A doughnut, 10 meters across, is more complex than a square 200 meters across, because a doughnut's edge is comprised of many small corners, and it has a subpolygon for the hole in the middle.

Where possible, keep polygons as simple as possible. There is no easy measure for determining the complexity of a polygon. If you find a polygon which is long, thin, has many corners, branches or holes, try to get the polygon smoothed out, or cut up into smaller polygons with fewer corners and subpolygons.

Excel sheets

The engine's calculations are optimized greatly, but with the ability to add your own excel sheets it's possible to add more complex calculations to the engine's tasks. If these calculations are not optimized as well, the engine will spend more time to perform these calculations. When designing your own calculation sheets, there are a number precautions you can take to prevent the engine slowing down too much on them. This can seem tricky, because when opened in Excel a sheet may seem to calculate quite quickly. However, when a different application, such as the Tygron Engine, handles an excel sheet, the calculations will take a bit longer. Under some circumstances, these small increases in calculation time can add up.

For debugging purposes, the engine displays the calculation time for excel indicators when they are selected in the editor.

Complex excel formulas

Excel is a powerful tool because of the functions and formulas it provides. However, some of the functions excel provides are more complex than others. For example, functions such as MATCH, which look up an value in a list by comparing it to another value, are more complex than INDEX, which simply returns the value in a given place in a list. It's not an issue to use a computationally heavy function here and there, but where possible they should be used as little as possible. Functions like IF and SUMIF should be avoided as well where possible, though they are not as problematic.

One concrete example is that often MATCH and INDEX are used together to look up a value and return something on the same row or column. If multiple values are to be retrieved from that same row, rather than using MATCH multiple times to determine the row number, use it once and store the row number in a separate cell. Then every time you need to retrieve values from that row, refer to that cell.

In general, avoid comparative lookups in excel sheets. One effective rule of thumb is to make your calculations as linear as possible. I.e. For a calculation of any given area, place all retrieved data and the calculations on a single row, moving from left to right as the calculation progresses.

X-queries

TQL offers the ability to create X-queries; queries in which the referred ID for a single item is replaced by an X, indicating that you wish to apply that query for all valid IDs. When an excel sheet is uploaded with an X-query, the engine prepares the excel sheet by transforming the single X-query into all the implied queries. This means that a single query can lead to hundreds of queries, if there are hundreds of valid IDs. Depending on how many X-queries are added to a single excel sheet, it's possible to quickly cause the sheet to perform thousands of queries.

When using X-queries, make sure to use as few as possible. Do not request data which you do not use. Additionally, when you find that you really only need a few of the many queries, and you can predict or control which ones they are, it can help greatly to replace the X-queries with regular queries.

Overlays

A few overlays in the engine are actually calculation models. These include, among others, the overlays for livability, subsidence, heat, and traffic noise. These calculations happen in the form of a grid; parts of which are updated whenever their underlying data is changed. The size of the cells of those grids can be changed. The smaller the cells, the more accurate the calculations and the shapes generated by the overlay. However, smaller cells mean more calculations. Selecting a smaller cell size means the engine will spend a greater amount of time to recalculate (parts of) an overlay.

Where possible, set the grid size of calculated as low as is required for your use-case but no lower.