.. _integrator_objectstorage: Working with Object storage (like S3) ===================================== Prepare files ------------- We can prepare a GeoTIFF for the Cloud, see the `COG file format `_ and the `GDAL output driver options `_. Generalities ------------ In this section, we explain how to use the S3-like storage from Exoscale. First of all, you should set the following variables on the ``geoportal``, ``config`` and ``qgisserver`` services: For Exoscale: * ``AWS_ACCESS_KEY_ID``: The project access key. * ``AWS_SECRET_ACCESS_KEY``: The project secret key. * ``AWS_DEFAULT_REGION=ch-dk-2``: The region used by Exoscale. * ``AWS_S3_ENDPOINT=sos-ch-dk-2.exo.io``: The endpoint used by Exoscale. For Azure: * ``AZURE_STORAGE_CONNECTION_STRING``: The connection string. For better performance, you should furthermore set the following variables on the ``geoportal`` and ``qgisserver`` services: * ``CPL_VSIL_CURL_USE_CACHE=TRUE`` * ``CPL_VSIL_CURL_CACHE_SIZE=128000000`` * ``CPL_VSIL_CURL_USE_HEAD=FALSE`` * ``GDAL_DISABLE_READDIR_ON_OPEN=TRUE`` Exoscale: Use the aws client to list the files: .. prompt:: bash aws --endpoint-url https://sos-ch-dk-2.exo.io/ --region ch-dk-2 \ s3 ls s3:/// Create the vrt file for a raster layer: .. prompt:: bash docker compose exec geoportal bash -c \ 'gdalbuildvrt /vsis3///index.vrt \ $(list4vrt / .tif)' Azure: Use list the container: .. prompt:: bash docker compose exec geoportal azure --list-container Use list the files: .. prompt:: bash docker compose exec geoportal azure --container= --list '' Create the vrt file for a raster layer: .. prompt:: bash docker compose exec geoportal azure --container= --vrt / .tiff MapServer --------- Create the shape index file for a raster layer: Exoscale: .. prompt:: bash docker compose exec geoportal bash -c \ 'gdaltindex mapserver/index.shp $( \ aws --endpoint-url http://${AWS_S3_ENDPOINT} \ --region ${AWS_DEFAULT_REGION} \ s3 ls s3://// | \ grep tif$ | \ awk '"'"'{print "/vsis3///"$4}'"'"' \ )' docker cp _geoportal_1:/app/index.shp mapserver/ docker cp _geoportal_1:/app/index.shx mapserver/ docker cp _geoportal_1:/app/index.dbf mapserver/ docker cp _geoportal_1:/app/index.prj mapserver/ Azure: .. prompt:: bash docker compose exec geoportal rm index.shp docker compose exec geoportal rm index.shx docker compose exec geoportal rm index.dbf docker compose exec geoportal rm index.prj docker compose exec geoportal bash -c \ 'gdaltindex mapserver/index.shp $( \ azure --container= --list / | \ grep tiff$ | \ awk '"'"'{print "/vsiaz//"$1}'"'"' \ )' docker cp _geoportal_1:/app/index.shp mapserver/.shp docker cp _geoportal_1:/app/index.shx mapserver/.shx docker cp _geoportal_1:/app/index.dbf mapserver/.dbf docker cp _geoportal_1:/app/index.prj mapserver/.prj Add the following config in the ``mapserver/mapserver.map.tmpl`` file: .. code:: CONFIG "CPL_VSIL_CURL_USE_CACHE" "TRUE" CONFIG "CPL_VSIL_CURL_CACHE_SIZE" "128000000" CONFIG "CPL_VSIL_CURL_USE_HEAD" "FALSE" CONFIG "GDAL_DISABLE_READDIR_ON_OPEN" "TRUE" Exoscale: .. code:: CONFIG "AWS_ACCESS_KEY_ID" "${AWS_ACCESS_KEY_ID}" CONFIG "AWS_SECRET_ACCESS_KEY" "${AWS_SECRET_ACCESS_KEY}" CONFIG "AWS_DEFAULT_REGION" "${AWS_DEFAULT_REGION}" CONFIG "AWS_S3_ENDPOINT" "${AWS_S3_ENDPOINT}" Azure: .. code:: ${DISABLE_LOCAL} CONFIG "AZURE_STORAGE_CONNECTION_STRING" "${AZURE_STORAGE_CONNECTION_STRING}" ${DISABLE_MUTUALIZE} CONFIG "AZURE_STORAGE_ACCOUNT" "${AZURE_STORAGE_ACCOUNT}" Use the shape index in the layer: .. code:: TYPE RASTER STATUS ON PROCESSING "RESAMPLE=AVERAGE" CONNECTIONTYPE OGR TILEINDEX "index.shp" TILEITEM "LOCATION" Add a vector layer for the object storage: .. code:: CONNECTIONTYPE OGR CONNECTION "${RASTER_BASE_PATH}/.shp" DATA "" `Some more information `_ .. note:: If you want to use different buckets or containers in different environments (such as integration / production), you should add an empty file named ``.raster`` (not ``.shp.raster``) and a ``RASTER_BASE_PATH`` environment variable in your `env.project` file, then the base path will be replaced (same number of folders). The empty raster files are here just to find the files that should be managed. Example: `RASTER_BASE_PATH=/vsiaz//` QGIS ---- Client ~~~~~~ The following environment variables should be defined (in the OS or in QGIS (``Settings`` / ``Options...`` / ``System`` / ``Environment``)): Exoscale: * ``AWS_ACCESS_KEY_ID``: The project access key. * ``AWS_SECRET_ACCESS_KEY``: The project secret key. * ``AWS_DEFAULT_REGION=ch-dk-2``: The region used by Exoscale. * ``AWS_S3_ENDPOINT=sos-ch-dk-2.exo.io``: The endpoint used by Exoscale. Azure: * ``AZURE_STORAGE_CONNECTION_STRING``: The connection string. On Windows also add: * ``GDAL_HTTP_UNSAFESSL=YES`` Then you can add a raster layer with: * Open from the menu ``Layer`` / ``Add Layer`` / ``Add Raster Layer``. * Section: ``Raster``. * ``Source type``: ``Protocole: HTTP(S), cloud, etc.``. * ``Type``: ``AWS S3`` or ``Microsoft Azure Blob``. * ``Bucket or container``: or . * ``Object key``: /index.vrt. You can add a vector layer in an analogous manner. Server ~~~~~~ Fill the required environment variables. Exoscale: * ``AWS_ACCESS_KEY_ID``: The project access key. * ``AWS_SECRET_ACCESS_KEY``: The project secret key. * ``AWS_DEFAULT_REGION=ch-dk-2``: Should already be in your env.project. * ``AWS_S3_ENDPOINT=sos-ch-dk-2.exo.io``: Should already be in your env.project. Azure docker compose: * ``AZURE_STORAGE_CONNECTION_STRING``: The connection string. For Azure AKS the access should be given by the AzureAssignedIdentity in Kubernetes, .. note:: If you want to use different buckets or containers in different environments (such as integration / production), you should add an empty file names ``.qgs.raster`` or ``.qgz.raster`` and a ``RASTER_BASE_PATH`` environment variable in your config container, then the base path will be replaced (same number of folder). The empty raster files are here just to find the files that should be managed.