README.md

Performance Testing Guide

3D Tiles Load Time Testing

When updating CesiumJS' 3D Tiles implementation, often it's helpful to ask "did this change help or hurt performance?". Here are some example scenarios where measuring the load time might be a helpful first step:

• Changing the tileset traversal algorithm may affect how efficiently tiles are loaded
• When 3D Tiles Next was implemented, we significantly refactored the glTF loading code. We started measuring performance to help identify potential problem spots.
• When evaluating compression algorithms, it's helpful to compare the load time of an uncompressed tileset with a compressed one.

This section is for measuring how long it takes a tileset and the initial view of tiles to load. This is a coarse measurement of performance; it is useful for quick comparisons, but not a replacement for using browser tools for performancing monitoring (such as Chrome's Performance tab).

The subsections below will explain the process for setting up a Sandcastle for performance testing, as in this Helsinki Example.

Pick a fixed camera view

For reproducibility, it is best to fix the camera in one spot so the same tiles load each time.

The code from this Sandcastle can be helpful to capture the camera parameters and print them to the console.

Load the tileset as you would normally, then manually adjust the camera to get a detailed view of the tileset. The goal is to load many tiles so differences in performance are more pronounced.

Helsinki data provided by Aalto University with support from City of Helsinki https://zenodo.org/record/5578198#.YjoTWBNKiu4

Once your view is configured, capture the parameters with the c key on the keyboard, which will result in a JS code snippet with the camera position baked in like this:

viewer.scene.camera.setView({
  destination: new Cesium.Cartesian3(
    2881774.230386411,
    1342620.6006856258,
    5510835.220814708,
  ),
  orientation: new Cesium.HeadingPitchRoll(
    5.708910165980631,
    -0.2293856922649915,
    2.875174018868165e-7,
  ),
});

Add the code to your final Sandcastle for performance testing.

Avoid extraneous requests

When creating the Viewer, turn off the globe and sky box, to avoid extra network requests that may impact the load time of your tileset.

const viewer = new Cesium.Viewer("cesiumContainer", {
  globe: false,
  skybox: false,
});

Load more detailed tiles

By default, Cesium sets the maximumScreenSpaceError threshold to 16 px. If the visible tile's screen space error goes above this threshold, tiles with more detail will be loaded.

For performance testing, we want a reasonably detailed scene without hitting out-of-memory problems or the test taking a long time. Decrease maximumScreenSpaceError as needed. For example:

tileset.maximumScreenSpaceError = 4;

How to measure elapsed time

In the browser, measuring elapsed time is quite simple using performance.now():

const start = performance.now();
expensiveOperation();
const end = performance.now();
const differenceSeconds = (end - start) / 1000.0;
console.log(`difference: ${differenceSeconds} sec`);

However, if you want to time multiple things, you might want to make yourself a small helper class. Here's an example:

// Helper class for measuring elapsed time.
class Timer {
  constructor(label) {
    this.label = label;
    this.startTime = undefined;
    this.endTime = undefined;
  }

  start() {
    this.startTime = performance.now();
  }

  stop() {
    this.endTime = performance.now();
  }

  print() {
    const difference_sec = (this.endTime - this.startTime) / 1000.0;
    console.log(`${this.label}: ${difference_sec} sec`);
  }
}

How to measure tileset load time

To time how long it takes for the tileset.json to be fetched and the Cesium3DTileset to be initialized (not including the tile content), start the timer just before creating the tileset with Cesium3DTileset.fromUrl, and stop it after the asynchronous function has completed.

tilesetTimer.start();
const tileset = await Cesium.Cesium3DTileset.fromUrl(url);
tilesetTimer.stop();
tilesetTimer.print();

How to measure tile load time

To time how long it takes for all the tiles in the current camera view to load (not including the tileset load time), start the timer after the tileset has been created with Cesium3DTileset.fromUrl and stop it in the initialTilesLoaded event handler. This event is used because we only care about our initial fixed camera view. allTilesLoaded, in contrast, may trigger multiple times if the camera moves, which is undesireable here.

const tileset = await Cesium.Cesium3DTileset.fromUrl(url);
tileTimer.start();

// This callback is fired the first time that all the visible tiles are loaded.
// It indicates the end of the test.
tileset.initialTilesLoaded.addEventListener(() => {
  tileTimer.stop();
  tileTimer.print();
});

Separating network time from code execution time

Depending on the scenario you are trying to test for, you may want to distinguish between the time used for network requests and code execution.

Quantifying network request time

For repeatability, you can throttle your connection speed to a fixed value, to eliminate network variations from time to time and from user to user. For example, in Chrome, see these instructions.

The speed you throttle to can be adjusted based on your target user base. For example, for tests targeting US users, you can throttle to the US median internet speed. Note that these speeds will change over time, so you will want to report the exact throttling speed as part of your test results.

Before throttling the connection, make sure to verify that your unthrottled network speed is at least as fast as the speed you are throttling to.

Quantifying code execution time

If you want to eliminate network latency and focus on measuring how long the initialization code takes to run, enable caching. In Chrome, this option can be done by unchecking the Disable Cache option in the Network tab. You will also need to enable caching on the server.

To check if caching is correctly enabled, load the tileset and check the Network tab. In Chrome, cached files will show up as (disk cache) in the Size column:

Loading tilesets with a static server

If using tilesets locally from disk, you will need a static server to load tilesets. Most any static server could work, though be sure to consider the following:

1. If caching is used, this must be enabled on the server. Note that the CesiumJS development server (npm run start) does not support caching.
2. If the static server is listening on a different domain or port, make sure to enable Cross-Origin Resource Sharing (CORS)

A example using the npm module http-server might look like this:

# Host the current directory at http://localhost:8003
# http-server defaults to caching with max-age of 1 hour.
# to disable caching, add the -c-1 flag
http-server -p 8003 --cors

Running the Test

When running the test, there are a few additional considerations:

1. Use the built version of Sandcastle to remove debug pragmas, combine JS files, and minify the results. This way, results reflect how users will be running Cesium.

   # Make a release build of CesiumJS. This step is necessary after switching
   # branches or otherwise modifying the code.
   npm run release
   
   # build Sandcastle and other apps
   npm run build-apps

2. (optional) Unless the Network tab is needed (e.g. for throttling), consider closing DevTools so the browser isn't spending time checking breakpoints or other performance monitoring tasks.

3. (If caching is used) warm up the cache. Run the Sandcastle once or twice and make sure all the tiles are loaded.

4. Do not adjust the camera or window size, so each run is consistent. Due to screen space error calculations, changing the window dimensions may cause different levels of detail to load.

5. Run the Sandcastle again to check the results. It may be desireable to run it multiple times and average the results.