TileGeneration

Introduction

With this solution, we solve the following issues:

  • Managing millions of files on the file system is difficult.

  • We should be able to update all the generated tiles.

  • We should not have thousands of expired files.

To do so, we need need a tool that can generate the tiles, update some of them contained in given geometries and delete empty tiles.

On-the-fly tiles generation introduces some issues such as having a growing number of tiles that may become unmanageable. For example, when updating the data, it is not possible to figure out what tiles should be updated.

For high usage websites, we want to put the tiles on S3 with the same tool.

One issue we have if we want to generate all the tiles is that the generation time can grow to more than one month, especially if we have a high resolution (low if in m/px) on the last zoom level. Therefore for the last zoom level we should generate the tiles on the fly with a low expiry (4 hours for example). Meta tiles should be used to reduce the number of requests to PostgreSQL. And the tiles should be deleted after the expiry time.

The chosen solution is a combination of two tools:

TileCloud-chain

TileCloud-chain is a TileCloud-based tool that offers a build chain for generating tiles from WMS of Mapnik on a local storage or S3 using a WMTS layout.

It supports the following AWS services for generating tiles: EC2, SQS, SNS.

See the readme.

Configuration

The configuration is done in the file tilegeneration/config.yaml.tmpl.

The main thing to do is to:

  • Set the resolutions we want to generate in the grids. If we want to generate different resolutions per layer, we should create different grids. Sub-level of grids is the grid name.

  • Configure the caches and set the generation/default_cache. Sub-level of caches is the cache name.

  • Configure the layer_default, the layers, and the generation/default_layers. Sub-level of layers is the layer name.

  • We can drop the empty tiles with a hash comparison, tilecloud-chain has a tool to help us:

    docker compose exec tilecloudchain generate-tiles \
   --get-hash <max-zoom>/0/0 --layer <layer>

    We consider that the first tile of the max zoom is empty. Then copy-paste the result in the layer config.

If you generate the tiles locally, you do not need all the configuration variables, because many of them in the generation part are for AWS generation.

Note

Points to check with TileCloud chain:

  • Disabling metatiles should be avoided.

  • Make sure that empty_metatile_detection and empty_tile_detection are configured correctly.

  • Make sure to not generate tiles with a resolution higher than the one in the raster sources.

Tile Generation and management

This package offers two tools, one to generate the tiles locally, see help:

docker compose exec tilecloudchain generate-tiles --help

one to generate the tiles using AWS, see help:

docker compose exec tilecloudchain generate_controller --help

Before starting a tile generation on S3, measure the cost:

docker compose exec tilecloudchain generate_controller --cost

If you setup all the default options, you can generate the tiles by using the command:

docker compose exec tilecloudchain generate-tiles

AWS credentials

To be able to connect to the S3 service, you should define the following variables in the env.project file:

AWS_ACCESS_KEY_ID=<access_key_id>
AWS_SECRET_ACCESS_KEY=<secret_access_key>

If you do not want to commit these credentials you can add them in your ~/.bashrc file:

export AWS_ACCESS_KEY_ID=<access_key_id>
export AWS_SECRET_ACCESS_KEY=<secret_access_key>

Web Interface

It is possible to run tile generation commands through a web interface located at URL <application main URL>/tiles/admin/. This interface require an authentication see Authentication for more information.

Predefined commands may be set in parameter server > predefined_commands in the tilegeneration/config.yaml.tmpl file.

See also