TileGeneration¶

Introduction¶

With this solution, we solve the following issues:

Managing millions of files on the file system is difficult.
We should be able to update all the generated tiles.
We should not have thousands of expired files.

To do so, we need need a tool that can generate the tiles, update some of them contained in given geometries and delete empty tiles.

On-the-fly tiles generation introduces some issues such as having a growing number of tiles that may become unmanageable. For example, when updating the data, it is not possible to figure out what tiles should be updated.

For high usage websites, we want to put the tiles on S3 with the same tool.

One issue we have if we want to generate all the tiles is that the generation time can grow to more than one month, especially if we have a high resolution (low if in m/px) on the last zoom level. Therefore for the last zoom level we should generate the tiles on the fly with a low expiry (4 hours for example). We should use metatiles to reduce the number of requests to postgres. And the tiles should be deleted after the expiry time.

The chosen solution is a combination of two tools:

TileCloud-Chain for the tile generation.

TileCloud-chain¶

TileCloud-chain is a TileCloud-based tool that offers a build chain for generating tiles from WMS of Mapnik on a local storage or S3 using a WMTS layout.

It supports the following AWS services for generating tiles: EC2, SQS, SNS.

See the readme.

Configuration¶

The configuration is done in the file tilegeneration/config.yaml.tmpl.

The main thing to do is to:

Set the resolutions we want to generate in the grids. If we want to generate different resolutions per layer, we should create different grids. Sub-level of grids is the grid name.
Configure the caches and set the generation/default_cache. Sub-level of caches is the cache name.
Configure the layer_default, the layers, and the generation/default_layers. Sub-level of layers is the layer name.
We can drop the empty tiles with a hash comparison, tilecloud-chain has a tool to help us:
```
docker-compose exec tilecloudchain generate_tiles \
   --get-hash <max-zoom>/0/0 --layer <layer>
```
We consider that the first tile of the max zoom is empty. Then copy-paste the result in the layer config.

If you generate the tiles locally, you do not need all the configuration variables, because many of them in the generation part are for AWS generation.

Note

Points to check with TileCloud chain:

Disabling metatiles should be avoided.
Make sure that empty_metatile_detection and empty_tile_detection are configured correctly.
Make sure to not generate tiles with a resolution higher than the one in the raster sources.

Tile Generation and management¶

This package offers two tools, one to generate the tiles locally, see help:

docker-compose exec tilecloudchain generate_tiles --help

one to generate the tiles using AWS, see help:

docker-compose exec tilecloudchain generate_controller --help

Before starting a tile generation on S3, measure the cost:

docker-compose exec tilecloudchain generate_controller --cost

If you setup all the default options, you can generate the tiles by using the command:

docker-compose exec tilecloudchain generate_tiles

AWS credentials¶

To be able to connect to the S3 service, you should define the following variables in the env.project file:

code:

AWS_ACCESS_KEY_ID=<access_key_id>
AWS_SECRET_ACCESS_KEY=<secret_access_key>

If you do not want to commit these credentials you can add them in your ~/.bashrc file:

code:

export AWS_ACCESS_KEY_ID=<access_key_id>
export AWS_SECRET_ACCESS_KEY=<secret_access_key>

Web Interface¶

It is possible to run tile generation commands through a web interface located at URL <application main URL>/tiles/admin/. This interface is protected by a password that must be specified in the C2C_SECRET environment variable in the env.project file.

Predefined commands may be set in parameter server > predefined_commands in the tilegeneration/config.yaml.tmpl file.

Upstream documentation¶

Configure¶

Configure grids¶

The grid describes how the tiles are arranged.

Especially on s3 or azure be careful to choose every of the grid settings before generating the tiles. If you change one of them you must regenerate all the tiles.

The resolutions in [px/m] describes all the resolutions available for this layer. For a raster layer, have a look to the maximum resolution of the source files. It is not needed to generate tiles at smaller resolutions than the sources, it is preferable to use the OpenLayers client zoom. Note that you can add a resolution in the end without regenerating all the tiles.

The bbox should match the resolution of the extent. CAREFUL: you will have big issue if you use this parameter to generate the tile on a restricted area: use the bbox on the layer instead.

The srs specifies the code of the projection.

The unit is the unit used by the projection.

The tile_size is the tile size in [px], defaults to 256.

The matrix_identifier is zoom by default and can also be set to resolution. It specifies how the z index is build to store the tiles, for example, for the resolutions [2, 1, 0.5] the used values are [0, 1, 2] based on the zoom and [2, 1, 0_5] based on the resolution. The second has the advantage of allowing to add a new resolution without regenerating all the tiles, but it does not work with MapCache.

Configure caches¶

The available tile caches are: s3, azure, bsddb, mbtile and filesystem.

The best solution to store the tiles, s3, azure, mbtiles and bsddb, have the advantage of using only one file per layer - style dimensions. To serve the mbtile and the bsddb see Distribute the tiles.

s3 needs a bucket and a folder (defaults to ‘’).

azure needs a container.

mbtiles, bsddb and filesystem just need a folder.

On all the caches we can add some information to generate the URL where the tiles are available. This is needed to generate the capabilities. We can specify:

http_url direct url to the tiles root.
http_urls (array) urls to the tiles root.
http_url and hosts (array), where each value of hosts is used to replace %(host)s in http_url.

In all case http_url or http_urls can include all attributes of this cache as %(attribute)s.

MBTiles vs Berkeley DB (`bsddb`)¶

Read performance: similar, eventually the MBTiles is 10% faster.
Write performance: The Berkeley DB is largely faster, about 10 times.
List the tiles: the MBTiles is largely faster, but we usually don’t need it.

Configure layers¶

First, all the attributes in layer_default are copied in all the layers to define the default values.

We have two type of layer: wms or mapnik.

To start the common attributes are:

min_resolution_seed the minimum resolution that is seeded, other resolutions are served by MapCache.

bbox used to limit the tiles generation.

px_buffer a buffer in px around the object area (geoms or extent).

WMTS layout¶

To generate the file paths and the WMTS capabilities we need additional information:

The mime_type of the tiles, it’s also used by the WMS GetMap and to upload the tiles.

The wmts_style defaults to ‘default’.

The extension is used to end the filename.

The dimensions (defaults to []) is an array of objects that have a name, a default value specified in the capabilities, a value to generate the tiles (it can be overwritten by an argument), and an array of values that contains all the possible values available in the capabilities.

For example if you generate the tiles and capabilities with the following configuration:

dimensions:
    -   name: DATE
        default: 2012
        value: 2012
        values: [2012]

then with the following configuration:

dimensions:
    -   name: DATE
        default: 2012
        value: 2013
        values: [2012, 2013]

We will have two set of tiles 2012 and 2013, both accessible by the capabilities, and by default we will see the first set of tiles.

Meta tiles¶

The meta tiles are activated by setting meta to on (by default it’s off).

The meta tiles are used for two things: first to generate multiple tiles with only one WMS query. By setting meta_size to 8 we will generate a square of 8 by 8 tiles in one shot.

The second usage of meta tiles is prevent cut label names: this is solved by getting a bigger image and cutting the borders. The meta_buffer should be set to a bigger value than half the size of the longest label.

Configure hash¶

We can filter tiles and meta tiles by using an hash.

The configuration of this hash is in the layer like this:

empty_metatile_detection:
    size: 740
    hash: 3237839c217b51b8a9644d596982f342f8041546
empty_tile_detection:
    size: 921
    hash: 1e3da153be87a493c4c71198366485f290cad43c

To easily generate this configuration we can use the following command:

generate-tiles --get-hash <z/x/y> -l <layer_name>

Where <z/x/y> should refer to an empty tile/metatile. Generally it’s a good idea to use z as the maximum zoom, x and y as 0.

Configure geom/sql¶

We can generate the tiles only on some geometries stored in PostGis.

The configuration is in the layer like this:

geoms:
-   connection: user=www-data password=www-data dbname=<db> host=localhost
    sql: <column> AS geom FROM <table>
    min_resolution: <resolution> # included, optional, last win
    max_resolution: <resolution> # included, optional, last win

Example:

geoms:
-   connection: user=postgres password=postgres dbname=tests host=localhost
    sql: the_geom AS geom FROM tests.polygon
-   connection: user=postgres password=postgres dbname=tests host=localhost
    sql: the_geom AS geom FROM tests.point
    min_resolution: 10
    max_resolution: 20

It’s preferable to use simple geometries, too complex geometries can slow down the generation.

Legends¶

To be able to generate legends with generate-controller --generate-legend-images you should have legend_mime and legend_extension in the layer configuration.

for example:

legend_mime: image/png
legend_extension: png

Then it will create a legend image per layer and per zoom level named .../1.0.0/{{layer}}/{{wmts_style}}/legend{{zoom}}.{{legend_extension}} only if she is different from the previous zoom level. If we have only one legend image it still stores in the file named legend0.{{legend_extension}}.

When we do generate-controller --generate-wmts-capabilities we will at first parse the legend images to generate a layer configuration like this:

legends:
-   mime_type: image/png
    href: http://host/tiles/layer/style/legend0.png
    min_resolution: 500 # optional, [m/px]
    max_resolution: 2000 # optional, [m/px]
    min_scale: # if define overwrite the min_resolution [m/m]
    max_scale: # if define overwrite the max_resolution [m/m]

If you define a legends array in the layer configuration it is directly used to generate the capabilities.

WMS layers¶

The additional value needed by the WMS is the URL of the server and the layers.

The previously defined mime_type is also used in the WMS requests.

To customize the request you also have the attributes params, headers and generate_salt. In params you can specify additional parameter of the WMS request, in headers you can modify the request headers. In version, you can change the WMS version. See the Proxy/cache issue for additional information.

Mapnik layers¶

We need to specify the mapfile path.

With Mapnik we have the possibility to specify a data_buffer then we should set the unneeded meta_buffer to 0.

And the output_format used for the Mapnik renderer, can be png, png256, jpeg, grid (grid_renderer).

Mapnik grid layers¶

With Mapnik we can generate UTFGrid tiles (JSON format that describes the tiles present on a corresponding tile) by using the output_format ‘grid’, see also: https://github.com/mapnik/mapnik/wiki/MapnikRenderers#grid_renderer.

Specific configuration:

We have a specific way to drop_empty_utfgrid by using the on value.

We should specify the pseudo pixel size [px] with the resolution.

And the layers_fields that we want to get the attributes. Object with the layer name as key and the values in an array as value.

In fact the Mapnik documentation says that’s working only for one layer.

And don’t forget to change the extension to json, and the mime_type to application/utfgrid and the meta to off (not supported).

Configuration example:

grid:
    type: mapnik
    mapfile: style.mapnik
    output_format: grid
    extension: json
    mime_type: application/utfgrid
    drop_empty_utfgrid: on
    resolution: 4
    meta: off
    data_buffer: 128
    layers_fields:
        buildings: [name, street]

Process¶

We can configure some tile commands to process the tiles. They can be automatically be called in the tile generation it we set the property post_process or pre_hash_post_process in the layer configuration.

The process is a set of names processes, and each one has a list of commands declared like this:

process:  # root process config
    optipng:  # the process command
    -   cmd: optipng %(args)s -q -zc9 -zm8 -zs3 -f5 -o %(out)s %(in)s  # the command line
        need_out: true  # if false the command rewrite the input file, default is false
        arg:  # argument used with the different log switches, in all cases default is ''
            default: '-q' # the argument used by default
            quiet: '-q' # the argument used in quiet mode
            verbose: '-v' # the argument used in verbose mode
            debug: '-log /tmp/optipng.log' # the argument user in debug mode

The cmd can have the following optional argument:

args the argument configured in the arg section.
in, out the input and output files.
x, y, z the tile coordinates.

Logging¶

Tile logs can be saved to a PostgreSQL database with this configuration:

..code:: yaml

logging:

database:
dbname: my_db host: db port: 5432 table: tilecloud_logs

PostgreSQL authentication can be specified with the PGUSER and PGPASSWORD environment variables. If the database is not reachable, the process will wait until it is.

Tiles error file¶

If we set a file path in configuration file:

generation:
    error_file: <path>

The tiles that’s in error will be append to the file, ant the tiles can be regenerated with generate-tiles --tiles <path>.

The <path> can be /tmp/error_{layer}_{datetime:%Y-%m-%d_%H:%M:%S} to have one file per layer and per run.

The tiles file looks like:

{.sourceCode .} # [time] some comments z/x/y # [time] the error z/x/y:+m/+m # [time] the error

The first line is just a comment, the second, is for an error on a tile, and the third is for an error on a meta tile.

Proxy/cache issue¶

In general we shouldn’t generate tiles throw a proxy, to do that you should configure the layers as this:

layers_name:
    url: http://localhost/wms
    headers:
        Host: the_host_name

The idea is to get the WMS server on localhost and use the Host header to select the right Apache VirtualHost.

To don’t have cache we use the as default the headers:

headers:
    Cache-Control: no-cache, no-store
    Pragma: no-cache

And if you steal have issue you can add a SALT random argument by setting the layer parameter generate_salt to true.

Alternate mime type¶

By default TileCloud support only the image/jpeg and image/png mime type.

Amazon services¶

Authentication¶

To be authenticated by Amazon you should set those environments variable before running a command:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

Configure S3¶

The cache configuration is like this:

s3:
    type: s3
    # the s3 bucket name
    bucket: tiles
    # the used folder in the bucket [default is '']
    folder: ''
    # for GetCapabilities
    http_url: https://%(host)s/%(bucket)s/%(folder)s/
    cache_control: 'public, max-age=14400'
    hosts:
    - wmts0.<host>

The bucket should already exists. If you don’t use Amazon’s S3, you must specify the host and the tiles_url configuration parameter.

Configure SQS¶

The configuration in layer is like this:

sqs:
    # The region where the SQS queue is
    region: eu-west-1
    # The SQS queue name, it should already exists
    queue: the_name

The queue should be used only by one layer.

To use the SQS queue we should first fill the queue:

generate-tiles --role master --layer <a_layer>

And then generate the tiles present in the SQS queue:

generate-tiles --role slave --layer <a_layer>

For the slave to keep listening when the queue is empty and be able to support more than one layer, you must enable the daemon mode and must not specify the layer:

generate-tiles --role slave --daemon

Configure SNS¶

SNS can be used to send a message when the generation ends.

The configuration is like this:

sns:
    topic: arn:aws:sns:eu-west-1:your-account-id:tilecloud
    region: eu-west-1

The topic should already exists.

Amazon tool¶

Amazon has a command line tool (homepage).

To use it, add in the setup.py:

awscli as an install_requires,
'aws = awscli.clidriver:main', in the console_scripts.

Than install it:

pip install awscli

And use it:

aws help

For example to delete many tiles do:

aws s3 rm --recursive s3://your_bucket_name/folder

Configure Azure¶

The cache configuration is like this:

azure:
    type: azure
    # the Azure container name
    container: tiles
    # the used folder in the container [default is '']
    folder: ''
    # for GetCapabilities
    http_url: https://%(host)s/%(bucket)s/%(folder)s/
    cache_control: 'public, max-age=14400'
    hosts:
    - wmts0.<host>

The container should already exists.

For the authentication you should set those environment variables: AZURE_STORAGE_CONNECTION_STRING on your local environment, or AZURE_STORAGE_ACCOUNT_URL if you run your container on Azure.

Configure the server¶

The server can be configure as it:

server:
    layers: a_layer # Restrict to serve an certain number of layers [default is all]
    cache: mbtiles # The used cache [default use generation/default_cache]
    # the URL without location to MapCache, [default is http://localhost/]
    geoms_redirect: true # use the geoms to redirect to MapCache [default is false]
    # allowed extension in the static path (default value), not used for s3.
    static_allow_extension: [jpeg, png, xml, js, html, css]

The minimal configuration is to enable it:

server: {}

You should also configure the http_url of the used cache, to something like https://%(host)s/${instanceid}/tiles or like https://%(host)s/${instanceid}/wsgi/tiles if you use the Pyramid view.

Pyramid view¶

To use the pyramid view use the following configuration:

config.get_settings().update({
    'tilegeneration_configfile': '<the configuration file>',
})
config.add_route('tiles', '/tiles/\*path')
config.add_view('tilecloud_chain.server:PyramidView', route_name='tiles')

Internal WSGI server¶

in production.ini:

[app:tiles]
use = egg:tilecloud_chain#server
configfile = %(here)s/tilegeneration/config.yaml

with the Apache configuration:

WSGIDaemonProcess tiles:${instanceid} display-name=%{GROUP} user=${modwsgi_user}
WSGIScriptAlias /${instanceid}/tiles ${directory}/apache/wmts.wsgi
<Location /${instanceid}/tiles>
    WSGIProcessGroup tiles:${instanceid}
    WSGIApplicationGroup %{GLOBAL}
</Location>

Commands¶

Available commands¶

generate-controller generate the annex files like legend.
generate-tiles generate the tiles.
generate-copy copy the tiles from a cache to an other.
generate-process process the tiles using a configured process.
generate-cost estimate the cost.
import_expiretiles import the osm2pgsql expire-tiles file as geoms in the database.

Each commands have a --help option to give a full arguments help.

Generate tiles¶

Generate all the tiles:

generate-tiles

Generate a specific layer:

generate-tiles --layer <a_layer>

Generate a specific zoom:

generate-tiles --zoom 5

Generate a specific zoom range:

generate-tiles --zoom 2-8

Generate a specific some zoom levels:

generate-tiles --zoom 2,4,7

Generate tiles from an (error) tiles file:

generate-tiles --layer <a_layer> --tiles <z/x/y>

Generate tiles on a bbox:

generate-tiles --bbox <MINX> <MINY> <MAXX> <MAXY>

Generate a tiles near a tile coordinate (useful for test):

generate-tiles --near <X> <Y>

Generate a tiles in a different cache than the default one:

generate-tiles --cache <a_cache>

Explain cost¶

Configuration (default values):

cost:
    # [nb/month]
    request_per_layers: 10000000
    cloudfront:
        download: 0.12,
        get: 0.009
    request_per_layers: 10000000
    s3:
        download: 0.12,
        get: 0.01,
        put: 0.01,
        storage: 0.125
    sqs:
        request: 0.01

Layer configuration (default values):

cost:
    metatile_generation_time: 30.0,
    tile_generation_time: 30.0,
    tile_size: 20.0,
    tileonly_generation_time: 60.0

The following commands can be used to know the time and cost to do generation:

generate-controller --cost

Useful options¶

--quiet or -q: used to display only errors.

--verbose or -v: used to display info messages.

--debug or -d: used to display debug message, please use this option to report issue. With the debug mode we don’t catch exceptions, and we don’t log time messages.

--test <n> or -t <n>: used to generate only <n> tiles, useful for test.

Mutualized¶

The mutualized mode consist by having multiple project files with the projects related configurations (layers, cache, …) and one main configuration file with the global configuration (number of process, log format, redis, …).

Configuration keys which should be set in the main configuration file are identified in property’s descriptions of the schema.json file.

Important remarks¶

Especially on S3 the grid name, the layer name, the dimensions, can’t be changed (understand if we want to change them we should regenerate all the tiles).

By default we also can’t insert a zoom level, if you think that you need it we can set the grid property matrix_identifier: resolution, bit it don’t work with MapCache.

Please use the --debug to report issue.

Environment variables¶

TILEGENERATION_CONFIGFILE: Default to /etc/tilegeneration/config.yaml, the all in one configuration file to use.
TILEGENERATION_MAIN_CONFIGFILE: Default to /etc/tilegeneration/config.yaml, the main configuration file to use.
TILEGENERATION_HOSTSFILE: Default to /etc/tilegeneration/hosts.yaml, the hosts to config file mapping file to use.
TILE_NB_THREAD: Default is 2, the number of thread used to generate the tiles (If we use meta tiles)
METATILE_NB_THREAD: Default is 25, the number of thread used to generate the meta tiles (If we use meta tiles, also to generate the tiles)
SERVER_NB_THREAD: Default to 10, the number of thread used to generate the meta tiles in the server.
TILE_QUEUE_SIZE: Default to 2, the queue size just after the Redis queue
TILE_CHUNK_SIZE: Default to 1, the chunk size to process the tiles after the meta tiles.
TILECLOUD_CHAIN_MAX_OUTPUT_LENGTH: Default to 1000, the maximum number of character of the output to be display in the admin interface.
LOG_TYPE: Default to console, can also be json to log in JSON for Logstash.
TILECLOUD_CHAIN_LOG_LEVEL Default to INFO, TILECLOUD_LOG_LEVEL Default to INFO, C2CWSGI_LOG_LEVEL Default to WARN, OTHER_LOG_LEVEL Default to WARN, the logging level of deferent components, can be DEBUG, INFO, WARN, ERROR or CRITICAL.
TILE_SERVER_LOGLEVEL Default to quiet the log level used in the server part.
TILE_MAPCACHE_LOGLEVEL``Default to ``verbose the log level used in the internal mapcache.
DEVELOPMENT: Default to 0 set it to 1 to have the Pyramid development options.
VISIBLE_ENTRY_POINT Default to /tiles/ the entrypoint path.

Admin and test pages¶

On the URL <base URL>/admin/ you can see the status of the generation, a tool to generate the tiles, and a link to a test page.

Beware, the test page assumes we have configured only one grid.

TileCloud-chain configuration¶

https://raw.githubusercontent.com/camptocamp/tilecloud-chain/master/tilecloud_chain/schema.json
type	object
properties
defaults	Defaults
	Used to put YAML references
	type	object
grids	Grids
	The WMTS grid definitions by grid name, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-grids
	type	object
	additionalProperties	Grid
caches	Caches
	The tiles caches definitions by name, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-caches
	type	object
	additionalProperties	Cache
layers	Layers
	The layers definitions by name, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-layers
	type	object
	additionalProperties	Layer
process	Process
	List of available commands by name, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#process
	type	object
	additionalProperties	Process command
generation	Generation
sqs	SQS
	The Simple Queue Service configuration
	type	object
	properties
	queue	Queue
		The queue name, default is ‘tilecloud’
		type	string
	region	AWS region
	additionalProperties	False
sns	SNS
	The Simple Notification Service configuration, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-sns
	type	object
	properties
	topic	Topic
		The topic
		type	string
	region	AWS region
	additionalProperties	False
redis	Redis
openlayers	OpenLayers
	Configuration used to generate the OpenLayers example page
	type	object
	properties
	srs	SRS
		The projection code
		type	string
		pattern	^EPSG:[0-9]+$
		default	EPSG:2056
	proj4js_def	The proj4js definition
		The proj4js definition, get it from https://epsg.io/
		type	string
		default	+proj=somerc +lat_0=46.95240555555556 +lon_0=7.439583333333333 +k_0=1 +x_0=2600000 +y_0=1200000 +ellps=bessel +towgs84=674.374,15.056,405.346,0,0,0,0 +units=m +no_defs
	center_x	Center x
		The center easting
		type	number
		default	2600000
	center_y	Center y
		The center northing
		type	number
		default	1200000
	zoom	Map initial zoom
		The initial zoom
		type	number
		default	3
	additionalProperties	False
server	Server
cost	Cost
metadata	Metadata
provider	Provider
logging	`Logging`_
authentication	Authentication
	The authentication configuration
	type	object
	properties
	github_repository	GitHub repository
		The GitHub repository name, on witch one we will check the access rights
		type	string
	github_access	GitHub access
		The kind of rights the user should have on the repository
		type	string
		enum	push, pull, admin
		default	pull
	additionalProperties	False
additionalProperties	False

Headers¶

The headers that we send to the WMS backend

type	object
additionalProperties	The header value
	type	string

Grid¶

The WMTS grid definition, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-grids

type	object
properties
resolution_scale	Resolution scale
	The scale used to build a FreeTileGrid typically ‘2’
	type	integer
resolutions	Resolutions
	The resolutions in pixel per meter
	type	array
	items	type	number
bbox	Bounding box
	The bounding box in meter
	type	array
	maxLength	4
	minLength	4
	items	type	number
srs	SRS
	The projection reference
	type	string
	pattern	^EPSG:[0-9]+$
proj4_literal	Proj4 literal
	The Proj4 definition
	type	string
unit	Unit
	The projection unit, default is ‘m’
	type	string
	default	m
tile_size	Tile size
	The tile size in pixel, default is 256
	type	integer
	default	256
matrix_identifier	Matrix identifier
	The identifier to use in the tiles URL, recommend to be resolution (default)
	type	string
	enum	zoom, resolution
	default	zoom
additionalProperties	False

Cache WMST capabilities file¶

The generated WMTS capabilities file name (by ‘default 1.0.0/WMTSCapabilities.xml’)

type

string

Cache HTTP URL¶

The HTTP URL %host will be replaces by one of the hosts

type

string

Cache host¶

The host used to build the HTTP URLs

type	array
items	type	string

Cache HTTP URLs¶

type	array
items	type	string

Cache folder¶

The root folder of the cache, default is ‘’

type	string
default

Cache filesystem¶

type	object
properties
type	const	filesystem
wmtscapabilities_file	Cache WMST capabilities file
http_url	Cache HTTP URL
hosts	Cache host
http_urls	Cache HTTP URLs
folder	Cache folder
additionalProperties	type	string

Cache S3¶

type	object
properties
type	const	s3
wmtscapabilities_file	Cache WMST capabilities file
http_url	Cache HTTP URL
hosts	Cache host
http_urls	Cache HTTP URLs
tiles_url	Tiles URL
	The template tiles URL on S3, the argument can be region, bucket and folder (default is ‘http://s3-{region}.amazonaws.com/{bucket}/{folder}’)
	type	string
host	Host
	The S3 host, default is ‘s3-eu-west-1.amazonaws.com’
	type	string
bucket	Bucket
	The S3 bucker name
	type	string
region	AWS region
cache_control	Cache control
	The Cache-Control used to store tiles on S3
	type	string
folder	Cache folder
additionalProperties	type	string

Cache Azure¶

Azure Blob Storage

type	object
properties
type	const	azure
wmtscapabilities_file	Cache WMST capabilities file
http_url	Cache HTTP URL
hosts	Cache host
http_urls	Cache HTTP URLs
folder	Cache folder
container	Container
	The Azure container name
	type	string
cache_control	Cache control
	The Cache-Control used to store tiles on Azure
	type	string
additionalProperties	type	string

Cache MBtiles¶

type	object
properties
type	const	mbtiles
wmtscapabilities_file	Cache WMST capabilities file
http_url	Cache HTTP URL
hosts	Cache host
http_urls	Cache HTTP URLs
folder	Cache folder
additionalProperties	type	string

Cache BSDDB¶

type	object
properties
type	const	bsddb
wmtscapabilities_file	Cache WMST capabilities file
http_url	Cache HTTP URL
hosts	Cache host
http_urls	Cache HTTP URLs
folder	Cache folder
additionalProperties	type	string

Cache¶

The tiles cache definition, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-caches

anyOf	Cache filesystem
	Cache S3
	Cache Azure
	Cache MBtiles
	Cache BSDDB

Layer title¶

The title, use to generate the capabilities

type

string

Layer grid¶

The used grid name

type

string

Layer bounding box¶

The bounding box where we will generate the tiles

type	array
maxLength	4
minLength	4
items	type	number

layer min resolution seed¶

The minimum resolutions to pre-generate

type

number

Layer pixel buffer¶

The buffer in pixel used to calculate geometry intersection, default is 0

type	integer
default	0

Layer meta¶

Use meta-tiles, default is False, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#meta-tiles

type	boolean
default	False

Layer meta size¶

The meta-tile size in tiles, default is 5

type

integer

Layer meta buffer¶

The meta-tiles buffer in pixels, default is 128

type

integer

Layer layers¶

The WMS layers

type

string

Layer WMTS style¶

The WMTS style

type	string
pattern	^[a-zA-Z0-9_\-\+~\.]+$

Layer MIME type¶

The MIME type of the tiles

type	string
pattern	^[a-zA-Z0-9!#$%^&\_\-\+{}\\|’.`~]+/[a-zA-Z0-9!#$%^&\_\-\+{}\\|’.`~]+$

Layer extension¶

The layer extension

type

string

Layer dimension name¶

The dimension name

type	string
pattern	^(?!(?i)(SERVICE\|VERSION\|REQUEST\|LAYERS\|STYLES\|SRS\|CRS\|BBOX\|WIDTH\|HEIGHT\|FORMAT\|BGCOLOR\|TRANSPARENT\|SLD\|EXCEPTIONS\|SALT))[a-z0-9_\-~\.]+$

layer dimensions¶

The WMTS dimensions

type	array
items	type	object
	properties
	name	Layer dimension name
	generate	Generate
		The values that should be generate
		type	array
		items	type	string
			pattern	^[a-zA-Z0-9_\-\+~\.]+$
	values	Values
		The values present in the capabilities
		type	array
		items	type	string
			pattern	^[a-zA-Z0-9_\-\+~\.]+$
	default	Default
		The default value present in the capabilities
		type	string
		pattern	^[a-zA-Z0-9_\-\+~\.]+$
	additionalProperties	False

Layer legends¶

The provided legend

type	array
items	type	object
	properties
	mime_type	MIME type
		The mime type used in the WMS request
		type	string
		pattern	^[a-zA-Z0-9!#$%^&\_\-\+{}\\|’.`~]+/[a-zA-Z0-9!#$%^&\_\-\+{}\\|’.`~]+$
	href	Href
		The URL of the legend image
		type	string
	width	Width
		The width of the legend image
		type	string
	height	Height
		The height of the legend image
		type	string
	min_scale	Min scale
		The max scale of the legend image
		type	string
	max_scale	Max scale
		The max scale of the legend image
		type	string
	min_resolution	Min resolution
		The max resolution of the legend image
		type	string
	max_resolution	Max resolution
		The max resolution of the legend image
		type	string
	additionalProperties	False

Layer legend MIME¶

The mime type used to store the generated legend

type	string
pattern	^[a-zA-Z0-9!#$%^&\_\-\+{}\\|’.`~]+/[a-zA-Z0-9!#$%^&\_\-\+{}\\|’.`~]+$

Layer legend extension¶

The extension used to store the generated legend

type	string
pattern	^[a-zA-Z0-9]+$

Layer pre hash post process¶

Do an image post process before the empty hash check

type

string

Layer post process¶

Do an image post process after the empty hash check

type

string

Layer geometries¶

The geometries used to determine where we should create the tiles, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-geomsql

type	array
items	type	object
	properties
	connection	Connection
		The PostgreSQL connection string
		type	string
	sql	SQL
		The SQL query that get the geometry in geom e.g. ‘the_geom AS geom FROM my_table’
		type	string
	min_resolution	Min resolution
		The min resolution where the query is valid
		type	number
	max_resolution	Max resolution
		The max resolution where the query is valid
		type	number
	additionalProperties	False

Layer empty tile detection¶

The rules used to detect the empty tiles, use generate-tiles –get-hash to get what we can use, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-hash

type	object
properties
size	Title
	The tile size
	type	integer
hash	Hash
	The tile hash
	type	string
additionalProperties	False

Layer empty meta-tile detection¶

The rules used to detect the empty meta-tiles, use generate-tiles –get-hash to get what we can use, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-hash

type	object
properties
size	Size
	The meta-tile size
	type	integer
hash	Hash
	The meta-tile hash
	type	string
additionalProperties	False

Layer cost¶

The rules used to calculate the cost

type	object
properties
tileonly_generation_time	tile only generation time
	The time to generate a tile without meta-tile
	type	number
	default	40
tile_generation_time	tile generation time
	The time to generate a tile from the meta-tile
	type	number
	default	30
metatile_generation_time	Meta tile generation time
	The time to generate a meta-tile
	type	number
	default	30
tile_size	Tile size
	The tile mean size in bytes
	type	number
	default	20
additionalProperties	False

Layer WMS¶

type	object
properties
type	const	wms
title	Layer title
grid	Layer grid
bbox	Layer bounding box
min_resolution_seed	layer min resolution seed
px_buffer	Layer pixel buffer
meta	Layer meta
meta_size	default	5
	Layer meta size
meta_buffer	default	128
	Layer meta buffer
layers	Layer layers
wmts_style	Layer WMTS style
mime_type	Layer MIME type
extension	Layer extension
dimensions	layer dimensions
legends	Layer legends
legend_mime	Layer legend MIME
legend_extension	Layer legend extension
pre_hash_post_process	Layer pre hash post process
post_process	Layer post process
geoms	Layer geometries
empty_tile_detection	Layer empty tile detection
empty_metatile_detection	Layer empty meta-tile detection
cost	Layer cost
url	URL
	The WMS service URL
	type	string
generate_salt	Generate salt
	Should generate a salt to drop the cache, default is False
	type	boolean
query_layers	Query layers
	The layers use for query (To be used with the server)
	type	string
info_formats	Info formats
	The query info format
	type	array
	items	type	string
params	Parameters
	Additional parameters to the WMS query (like dimension)
	type	object
	additionalProperties	The parameter value
		type	string
headers	Headers
version	Version
	The used WMS version (default is ‘1.1.1’)
	type	string
additionalProperties	False

Layer Mapnik¶

type	object
properties
type	const	mapnik
title	Layer title
grid	Layer grid
bbox	Layer bounding box
min_resolution_seed	layer min resolution seed
px_buffer	Layer pixel buffer
meta	Layer meta
meta_size	default	1
	Layer meta size
meta_buffer	default	0
	Layer meta buffer
layers	default	__all__
	Layer layers
wmts_style	Layer WMTS style
mime_type	Layer MIME type
extension	Layer extension
dimensions	layer dimensions
legends	Layer legends
legend_mime	Layer legend MIME
legend_extension	Layer legend extension
pre_hash_post_process	Layer pre hash post process
post_process	Layer post process
geoms	Layer geometries
empty_tile_detection	Layer empty tile detection
empty_metatile_detection	Layer empty meta-tile detection
cost	Layer cost
mapfile	MapFile
	The Mapnik map file
	type	string
data_buffer	Data buffer
	The data buffer, default is 128
	type	integer
output_format	Output format
	The Mapnik output format, default is ‘png’
	type	string
	enum	png, png256, jpeg, grid
wms_url	WMS URL
	A WMS fallback URL (deprecated)
	type	string
resolution	Resolution
	The resolution, default is 4
	type	integer
layers_fields	Layers fields
	The Mapnik layers fields
	type	object
	additionalProperties	The Mapnik layer fields
		type	array
		items	type	string
drop_empty_utfgrid	Drop empty UTFGrid
	Drop if the tile is empty, default is False
	type	boolean
additionalProperties	False

Layer¶

The layer definition, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-layers

anyOf	layer_wms
	layer_mapnik

Process command¶

A command

type	array
items	type	object
	properties
	cmd	Command
		The shell command, available parameters: %(in)s, %(out)s, %(args)s, %(x)s, %(y)s, %(z)s.
		type	string
	need_out	Need out
		The command will generate an output in a file, default is False
		type	boolean
		default	False
	arg	Argument
		Used to build the %(args)
		type	object
		properties
		default	Properties
			The arguments used by default
			type	string
		verbose	Verbose
			The arguments used on verbose mode
			type	string
		debug	Debug
			The arguments used on debug mode
			type	string
		quiet	Quiet
			The arguments used on quiet mode
			type	string
		additionalProperties	False
	additionalProperties	False

Generation¶

The configuration used for the generation

type	object
properties
default_cache	Default cache
	The default cache name to be used, default do ‘default’
	type	string
	default	default
default_layers	Default layers
	The default layers to be generated
	type	array
	items	type	string
authorised_user	Authorised user
	The authorised user to generate the tiles (used to avoid permission issue on generated tiles) (main configuration)
	type	string
maxconsecutive_errors	Max consecutive errors
	The maximum number of consecutive errors (main configuration), default is 10
	type	integer
	default	10
error_file	Error file
	File name generated with the tiles in error, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#tiles-error-file (main configuration)
	type	string
number_process	Number process
	Number of process used to generate the tiles (main configuration), default is 1
	type	integer
	default	1
additionalProperties	False

AWS region¶

The region, default is ‘eu-west-1’

type	string
pattern	^(eu\|us\|ap\|sa)-(north\|south\|east\|west\|central)(east\|west)?-[1-3]$

Redis¶

The Redis configuration (main configuration)

type	object
properties
url	URL
	The server URL (main configuration)
	type	string
	pattern	^redis://[^:]+:[^:]+$
sentinels	Sentinels
	The sentinels (main configuration)
	type	array
	items	A sentinel (main configuration)
		type	array
		items
			Sentinel host
			The sentinel host name (main configuration)
			type	string
			Sentinel port
			The sentinel port (main configuration)
			anyOf	type	string
				type	integer
		additionalItems	False
connection_kwargs	The Redis connection arguments (main configuration)
	type	object
sentinel_kwargs	The Redis sentinel arguments (main configuration)
	type	object
service_name	Service name
	The service name, default is ‘mymaster’ (main configuration)
	type	string
	default	mymaster
socket_timeout	Socket timeout
	The socket timeout (main configuration)
	type	integer
db	Database
	type	integer
queue	Queue
	The queue name (main configuration)
	type	string
	default	tilecloud
timeout	Timeout
	The timeout (main configuration), default is 5
	type	integer
	default	5
pending_timeout	Pending timeout
	The pending timeout (main configuration), default is 300
	type	integer
	default	300
max_retries	Max retries
	The max retries (main configuration), default is 5
	type	integer
	default	5
max_errors_age	Max errors age
	The max error age (main configuration), default is 86400 (1 day)
	type	integer
	default	86400
max_errors_nb	Max errors number
	The max error number (main configuration), default is 100
	type	integer
	default	100
prefix	Prefix
	The prefix (main configuration), default is ‘tilecloud_cache’
	type	string
	default	tilecloud_cache
expiration	Expiration
	The meta-tile in queue expiration (main configuration), default is 28800 (8 hours)
	type	integer
	default	28800
additionalProperties	False

Server¶

Configuration used by the tile server, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#distribute-the-tiles

type	object
properties
cache	Cache
	The used cache name
	type	string
layers	WMS Layers
	Layers available in the server, default is all layers
	type	array
	items	type	string
geoms_redirect	Geometries redirect
	Take care on the geometries, default is False
	type	boolean
	default	False
static_allow_extension	Static allow extension
	The allowed extension of static files, defaults to [jpeg, png, xml, js, html, css]
	type	array
	items	type	string
wmts_path	WMTS path
	The sub-path for the WMTS (main configuration), default is ‘wmts’
	type	string
	default	wmts
static_path	Static path
	The sub-path for the static files (main configuration), default is ‘static’
	type	string
	default	static
admin_path	Admin path
	The sub-path for the admin (main configuration), default is ‘admin’
	type	string
	default	admin
expires	Expires
	The browser cache expiration, default is 8 (hours)
	type	integer
	default	8
predefined_commands	Predefined commands
	The predefined commands used to generate the tiles
	type	array
	items	type	object
		properties
		command	Command
			The command to run
			type	string
		name	Name
			The name used in the admin interface
			type	string
		additionalProperties	False
allowed_commands	Allowed commands
	The allowed commands (main configuration)
	type	array
	items	type	string
allowed_arguments	Allowed arguments
	The allowed arguments (main configuration)
	type	array
	items	type	string
additionalProperties	False

Cost¶

The configuration use to calculate the cast (unmaintained)

type	object
properties
request_per_layers	Request per layers
	Tile request per hours, default is 10 000 000
	type	integer
	default	10000000
s3	S3 cost
	The S3 cost (main configuration)
	type	object
	properties
	storage	Storage
		The storage cost in $ / Gio / month (main configuration)
		type	number
		default	0.125
	put	Put
		The cost of put in $ per 10 000 requests (main configuration)
		type	number
		default	0.01
	get	Get
		The cost of get in $ per 10 000 requests (main configuration)
		type	number
		default	0.01
	download	Download
		The cost of download in $ per Gio (main configuration)
		type	number
		default	0.12
	additionalProperties	False
cloudfront	CloudFront cost
	The CloudFront cost (main configuration)
	type	object
	properties
	get	Get
		The cost of get in $ per 10 000 requests (main configuration)
		type	number
		default	0.009
	download	Download
		The cost of download in $ per Gio (main configuration)
		type	number
		default	0.12
	additionalProperties	False
sqs	SQS cost
	The SQS cost, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#configure-sqs (main configuration)
	type	object
	properties
	request	Request
		The cost of request in $ per 1 000 000 requests (main configuration)
		type	number
		default	0.01
	additionalProperties	False
additionalProperties	False

Metadata¶

The configuration of the WMTS capabilities metadata

type	object
properties
title	Title
	The title
	type	string
abstract	Abstract
	The abstract
	type	string
servicetype	Service type
	The service type, default is ‘OGC WMTS’
	type	string
	default	OGC WMTS
keywords	Keywords
	The keywords
	type	array
	items	type	string
fees	Fees
	The fees
	type	string
access_constraints	Access constraints
	The access constraints
	type	string
additionalProperties	False

Provider¶

The provider

type	object
properties
name	Name
	type	string
url	URL
	The public URL
	type	string
contact	Contact
	The contact
	type	object
	properties
	name	Name
		type	string
	position	Position
		type	string
	info	Info
		The information
		type	object
		properties
		phone	Phone
			The phone
			type	object
			properties
			voice	Voice
				The voice number
				type	string
			fax	Fax
				The fax number
				type	string
			additionalProperties	False
		address	Address
			The address
			type	object
			properties
			delivery	Delivery
				The delivery
				type	string
			city	City
				The city
				type	string
			area	Area
				The area
				type	string
			postal_code	Postal code
				The postal code
				type	integer
			country	Country
				The country
				type	string
			email	Email
				The email
				type	string
			additionalProperties	False
		additionalProperties	False
	additionalProperties	False
additionalProperties	False

Logging¶

The logging configuration to database, see https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst#logging (main configuration)

type	object
properties
database	Database
	The database (main configuration)
	type	object
	properties
	host	Host
		The host (main configuration)
		type	string
	port	Port
		The port (main configuration), default is 5432
		type	integer
		default	5432
	dbname	Database
		The database name (main configuration)
		type	string
	table	Table
		The table name (main configuration)
		type	string
	user	User
		The user name (main configuration)
		type	string
	password	Password
		The password (main configuration)
		type	string
	additionalProperties	False
additionalProperties	False

TileGeneration¶

Introduction¶

TileCloud-chain¶

Configuration¶

Tile Generation and management¶

AWS credentials¶

Web Interface¶

See also¶

Upstream documentation¶

Configure¶

Configure grids¶

Configure caches¶

MBTiles vs Berkeley DB (bsddb)¶

Configure layers¶

WMTS layout¶

Meta tiles¶

Configure hash¶

Configure geom/sql¶

Legends¶

WMS layers¶

Mapnik layers¶

Mapnik grid layers¶

Process¶

Logging¶

Tiles error file¶

Proxy/cache issue¶

Alternate mime type¶

Amazon services¶

Authentication¶

Configure S3¶

Configure SQS¶

Configure SNS¶

Amazon tool¶

Configure Azure¶

Other related configuration¶

Configure the server¶

Pyramid view¶

Internal WSGI server¶

Commands¶

Available commands¶

Generate tiles¶

Explain cost¶

Useful options¶

Mutualized¶

Important remarks¶

Environment variables¶

Admin and test pages¶

TileCloud-chain configuration¶

Headers¶

Grid¶

Cache WMST capabilities file¶

Cache HTTP URL¶

Cache host¶

Cache HTTP URLs¶

Cache folder¶

Cache filesystem¶

Cache S3¶

Cache Azure¶

Cache MBtiles¶

Cache BSDDB¶

Cache¶

Layer title¶

Layer grid¶

Layer bounding box¶

layer min resolution seed¶

Layer pixel buffer¶

Layer meta¶

Layer meta size¶

Layer meta buffer¶

Layer layers¶

Layer WMTS style¶

Layer MIME type¶

Layer extension¶

Layer dimension name¶

layer dimensions¶

Layer legends¶

Layer legend MIME¶

Layer legend extension¶

Layer pre hash post process¶

Layer post process¶

MBTiles vs Berkeley DB (`bsddb`)¶