Server-side development

Create development environment in a project

c2cgeoportal developers often need to test c2cgeoportal changes in the context of an existing c2cgeoportal application. Here is how:

  • Change current directory to your sources root directory and clone

    c2cgeoportal there (in the following line the sources directory is the user home directory). Then checkout the submodules, build c2cgeoportal, and go back to your project directory:

    cd ~
git clone git@github.com:camptocamp/c2cgeoportal.git
cd c2cgeoportal
git submodule update --init
./docker-run make build
cd  ~/<project>

Note

Old build directory

If you still have old build directory remove it before running the build

rm -rf .build

You can now check out your development branch if necessary.

  • Edit your <user>.mk to have something like this:

    INSTANCE_ID = <user>
DEVELOPMENT = TRUE
REQUIREMENTS = -e ../c2cgeoportal

include <package>.mk

  • Uninstall the regular c2cgeoportal egg from the virtual environment:

    .build/venv/bin/pip uninstall c2cgeoportal

  • Remove/comment the following line in the CONST_requirements.txt file:

    c2cgeoportal==1.6.0

  • Build your application application:

    rm .build/requirements.timestamp && make -f <user>.mk build

Note

Print performance issue

When restarting the print server frequently, performance issues may randomly be observed. This is done in random number generation.

To improve the performances you should add in the /srv/tomcat/tomcat1/bin/setenv-local.sh file:

export ADD_JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom"

Tests

Running tests

To be able to run c2cgeoportal tests you need to have the c2cgeoportal source code, and a make environment for it. So do that first, as described below.

Install c2cgeportal from source

Check out c2cgeoportal from GitHub:

git clone git@github.com:camptocamp/c2cgeoportal.git

Change into the c2cgeoportal directory and initialize the submodules:

cd c2cgeoportal
git submodule update --init

c2cgeoportal has two types of tests: unit tests and functional tests. The unit tests are self-contained, and do not require any specific setup. The functional tests require to run with docker-compose-run.

Unit tests

To run the unit tests do this:

./docker-run make build
docker build --tag=geomapfish-db docker/test-db
./docker-compose-run alembic --config c2cgeoportal/tests/functional/alembic.ini upgrade head
./docker-compose-run alembic --config c2cgeoportal/tests/functional/alembic_static.ini upgrade head
./docker-compose-run make tests

To run a specific test use the --where switch. For example:

./docker-compose-run nosetests --where \
    /src/c2cgeoportal/tests/functional/test_themes.py:TestThemesView.test_catalogue

Adding tests

To Be Done

Profiling

.build/venv/bin/pip install wsgi_lineprof

At the end of the file apache/application.wsgi add:

from wsgi_lineprof.middleware import LineProfilerMiddleware
from wsgi_lineprof.filters import FilenameFilter, TotalTimeSorter

filters = [
    FilenameFilter("entry.py"),
    TotalTimeSorter(),
]
application = LineProfilerMiddleware(application, stream=open('/tmp/pro', 'w'), filters=filters)

sudo apache2ctl graceful

Do your request(s).

The profile result will be in the file /tmp/pro.

Upgrade dependencies

When we start a new version of c2cgeoportal or just before a new development phase it is a good idea to update the dependencies.

Eggs

All the c2cgeoportal (and tilecloud-chain) dependencies are present in the c2cgeoportal/scaffolds/update/CONST_versions.mako file.

To update them you can simply get them from a travis build in the .build/venv/bin/pip freeze task.

Submodules

Go to the OpenLayers folder:

cd c2cgeoportal/static/lib/openlayers/

Get the new revision of OpenLayers:

git fetch
git checkout release-<version>

Then you can commit it:

cd -
git add c2cgeoportal/static/lib/openlayers/
git commit -m "update OpenLayers to <version>"

Database

Object model

../_images/database.png

TreeItem and TreeGroup are abstract (cannot be create) class used to create the tree.

FullTextSearch references a first level LayerGroup but without any constrains.

it is not visible on this schema, but the User of a child schema has a link (parent_role) to the Role of the parent schema.

metadata vs functionality

Technically the same functionality can be reused by more than one element.

functionalities are designed to configure and customize various parts of the application. For instance to change the default basemap when a new theme is loaded.

To do that in the CGXP application we trigger an event when we load a theme the new functionnalities.

The metadata contains attributes that are directly related to the element. For example the layer disclaimer, …

Migration

We use the alembic module for database migration. alembic works with a so-called migration repository, which is a simple directory in the application source tree:CONST_alembic. As the CONST_ prefix suggests this repository is part of the c2cgeoportal_update scaffold, it is created or updated when this scaffold is applied. So developers who modify the c2cgeoportal database schema should add migration scripts to the c2cgeoportal_update scaffold, as opposed to the application.

Add a new script call from the application’s root directory:

.build/venv/bin/alembic --config alembic[_static].ini revision --message "<Explicit name>"

Or in c2cgeoportal root directory:

.build/venv/bin/alembic \
    --config c2cgeoportal/tests/functional/alembic[_static].ini \
    revision --message "<Explicit name>"

This will generate the migration script in CONST_alembic/versions/xxx_<Explicite_name>.py You should NOT commit the script in this directory because this migration script should be shared with all c2cgeoportal projects. It is the c2cgeoportal update template which is responsible for updating this directory.

The script should not get any part of the c2cgeoportal.model module!

To get the project schema use: schema = context.get_context().config.get_main_option('schema')

The scripts should not fail if it is run again. See: http://alembic.readthedocs.org/en/latest/cookbook.html#conditional-migration-elements

Then customize the migration to suit your needs, test it:

.build/venv/bin/alembic upgrade head

Once you have tested it, move it to the c2cgeoportal update template, in c2cgeoportal/scaffolds/update/CONST_mlembic/versions/.

More information at:

Sub domain

All the static resources used sub domains by using the configurations variables: subdomain_url_template and subdomains.

To be able to use sub domain in a view we should configure the route as this:

from c2cgeoportal.lib import MultiDomainPregenerator
config.add_route(
    '<name>', '<path>',
    pregenerator=MultiDomainPregenerator())

And use the route_url with an additional argument subdomain:

request.route_url('<name>', subdomain='<subdomain>')}",

Code

Coding style

Please read http://www.python.org/dev/peps/pep-0008/.

And run validation:

make checks