Topographic Web Map of Mountainous Areas

3.1. Software installation

After installing Ubuntu, some dependencies should also be installed:

This list includes various utilities and libraries, the Apache web server, and “carto” which is used to convert Carto-CSS stylesheets into something that the map renderer can understand.

3.1.1. Installing postgresql / postgis

Postgresql is provides the database for the osm raw data. On Ubuntu, there are pre-packaged versions of both postgis and postgresql, so these can simply be installed via the Ubuntu package manager:

Here “postgresql” is the database for storing map data and “postgis” adds some extra graphical support to it. Now the postgis database needs to be created.On various tutorials, it is assumed that the name of the database is “gis”, so this name was used here also. It also requires a username and a password for the rendering account. The render account in this case is called “andreea”.

“Sudo” allows a system administrator to give certain users the ability to run commands as “root”, “-u” does the actual installation of whatever program the user wants to install. With the command above we created the account for rendering and a database named “gis”. UTF-8 is a compromise character encoding that can be as compact as ASCII (if the ile is plain English text), buut can also contain any unicode characters (with some increase in file size). UTF means Unicode Transformation Format and 8 means that it uses 8-bit block to represent a character.

The database needs to be set up: the extensions need to be created at first with the command

In postgis, it is required to set up the tables for the database: columns and rows, so the data for Romania will fit perfectly into the tables. The command is simple, a postgresql command, but in the Ubuntu command line environment:

and

3.1.2. Installing osm2pgsql

To import and manage the OpenStreetMap data into the database, osm2pgsql was used. This is a powerful command line-based program that actually converts raw osm data into the database.

The build mechanism used by osm2pgsql has changed since older versions, so there is a requirement to install more prerequisites for that:

Boost is a set of libraries for the C++ programming language that provide support for tasks and structures such as linear algebra, pseudorandom number generation, multithreading, image processing, regular expressions, and unit testing. It contains over eighty individual libraries.

The next command makes the installation of osm2pgsql:

3.1.3. Mapnik installation

Like already said in this paper, Mapnik is the library that does the actual rendering of the maps. There are many software libraries like Mapnik, but the authors chose this one because it is based on Python programming language and they are familiar with it. For this software, the latest version (Mapnik 2.2 – in May 2017 that was the latest version for Ubuntu) was used.

This command installs the libraries for Mapnik, based on apache2 libraries. The software works together with gdal tool, which also provides libraries and dependencies for it.

To Install the tool takes a lot of time (hours in our case) and brings back lots of messages to the command line, so it is very time consuming to check for errors through these messages. To check if the tool is install correctly, it can be imported in python. If this will result in an error, then Mapnik is not correctly installed. If Mapnik is correctly installed, no error will appear, which means that Python can import it:

3.1.4. Install mod_tile and renderd

Mod_tile is the system that serves raster tiles. Renderd actually renders the tiles when mod_tile requests them. The two software were downloaded from Github.

3.2. Stylesheet configuration

The style that was used is the one used by the standard maps on the Open Street Map official site, which was modified for the contour lines and hillshading. This stylesheet is well documented and works everywhere in the world; for these reasons, it was chosen.

The downloaded stylesheet is in MML format, but Mapnik reads only XML formats. For this reason, a compiler (carto) needs to convert the original file in something that Mapnik can understand:

The output from the command above is a Mapnik stylesheet named “mapnik.xml” in the same location as the input raw data.

3.3. Loading the data into the PostGis database

The raw data (coordinates for all amenities and landcover) was downloaded from GeoFabrik. Here one can find raw data for the whole world, which is divided in continent/country sections. The format available for download is *.osm.pbf or zipped shapefiles. For the project, we need the file named romania-latest.osm.pbf. At the starting time of the project, it was last updated on 15th of May 2017, but since then it was several times updated; Geofabrik updates its data once a week, on Wednesdays.
The data was downloaded in this way:

The next step is to load the data into the Postgis database, that was created in step 3.1. For this step, osm2pgsl is needed. This tool will import the data in the database, create way, nodes and relations out of the raw osm data. The following command was used in the Ubuntu command line:

The database to work with is “gis”, that was created before, when installing PostgreSQL (-d gis in the command). The data is loaded now into an empty database (-create). When importing other data in the same database, the command will be -append.
Osm2pgsql can use different table layouts, but for rendering only “slim” tables work. The -G command determines how the multipolygons are processed. To allow tags for which there are no explicit database columns to be used for rendering, the - -hstore command was used. The script used for the tag process is “openstreetmap-carto.lua”. It is defined with the command - - tag-transform-script. This is an easy way to process OSM tags before the style itself processes them, making the style logic potentially much simpler.

For the import process 2.5 GB of memory were allocated to osm2pgsql (-C 2500). The command - - number-processes 1 allocates 1 CPU to the importing process. -S means creating the database columns in the input file (*.pbf). Now the data is loading and it takes a while, depending on the computer used and on the amount of data that is imported. In this case it took over 6 hours to load the data, because the raw data was big (a whole country). It made over 10 000 ways, 3 000 nodes and 4 000 relations.

Some shape files are still needed for the low-zoom country boundaries. They were downloaded and indexed using a python script from openstreetmap-carto. It takes considerable time when running the script to download all the shapefiles for all countries.

3.4. Installing fonts

Around the world, not all the places are written in Latin characters, but because Romania is using the a-z alphabet, there is no need to install other fonts than the default. It may be that at some point other data from around the world will be inserted in the same database, in order to create the same kind of elevation maps that are now created for Romania, so the fonts were however installed:

3.5. Setting up the webserver

3.5.1. Renderd configuration

Renderd is the tool that renders the tiles so it needs a connection to the data that needs to be rendered, to the styles that we need to use and to a website where the tiles are displayed. It has a configuration file, where one should write the number of threads, the directory to which rendered tiles are going, the settings for Mapnik and an “ajt” section that makes the connection between the stylesheet and the rendering process (see Figure 1). This section actually corresponds to the named map style.

In this configuration file, the most important change that needs to be done is the location of the XML stylesheet. All the rest are set by default when installing Mapnik, Mod_tile and Renderd. The number of threads is set to 4 by default, but if the computer has less than 4 GB of memory (like in this case), the value should be changed to 2. If there are more styles on the map, one needs more map style sections on the configuration file of Renderd, but the URI should be different. In this case there is just one style section because only one style was used, with the URI “hot”, chosen so that the tiles generated can more easily be used in place of the HOT tile layer at OpenStreetMap.org. Of course, the URI name can be modified to choose another layer for the tile viewing.

3.5.2. Configuring Apache

In the configuration file of Apache2, a connection with Mod_tile needs to be made. The following line has been added in the file:

After this, the configuration file needs to be ran, so it can process the changed line from above:

Another connection needs to be made between Apache and Renderd. For this the virtual host configuration file of Apache was modified. Virtual host files are the files that specify the actual configuration of the virtual hosts and dictate how the Apache web server will respond to various domain requests. Apache comes with a default virtual host file called “000-default.conf”. To this file the following lines were added between the ServerAdmin section and Document lines, in order to “tell” Apache about Renderd:

After this, Apache needs to be reloaded twice:

Now, if a web browser is pointed to "http://server_ip_address/index.html" will result in the Ubuntu/Apache’s “It Works!” page (see Figure 2).

3.5.3. Running Renderd for the first time

In order to start rendering tiles, the configuration file of Renderd needs to be run, first of all because some changes were made to this file and secondly because it is the first time when the rendering process is taking place on the server since the installation:

***Note: this command needs to be done every time something is changing in the configuration file (e.g. when modifying the XML style sheet). In this way Renderd is ran in the foreground and all the errors (if there are any) will be shown on Ubuntu’s command line (e.g. errors from the style sheet). On a computer that has Ubuntu as the basic operating system, from here the rendering process starts without problems. On the other hand, if the server is set up on the virtual machine based on Ubuntu, one needs to reboot every time after running the above command. It might be because of the allocated resources to the virtual machine.

After this, if the browser is pointed at http://server_ip_address/hot/0/0/0.png, a map of the world is shown (see Figure 3).

3.5.4. Running renderd in the background and viewing tiles

To make this program start automatically every time, this should be set:

To view the tiles, there are 2 possibilities: create a website connected to the server or use a website like OpenStreetMap.org to view the rendered tiles. This is more like “borrowing” the OpenStreetMap website to be able to see your tiles. For the presentation of the project, the second option was used at first. In a subsequent phase, a webpage was created for displaying the tiles created.

"Borrowing openstreetmap.org"

In a Google Chrome browser “Switcheroo Redirector” extension was installed (source: https://chrome.google.com/webstore/detail/switcheroo-redirector) and set up to take the rendered tiles from the server and bring them to the layer “Humanitarian” from OpenStreetMap.org. At the section “From” http://tile-a.openstreetmap.fr/hot/ was added, and at the “To” section the IP address of the server was typed in (see Figure 4).

Now, when pointing the browser to Romania's coordinates and switch to the Humanitarian layer, a map of the world will be displayed and Romania is the only country that appears on this map. This happens because in the server database just the data from Romania exists. (see Figure 5).

In the background Renderd is doing the rendering process, so whenever the map is zoomed in with focus on Romania, it creates new tiles (see Figure 6 for an example).

3.5.5 Adding contour lines and hillshading to the server

The purpose of the project is to create beautiful maps from the server, with contour lines and 3D effect, which can be seen especially in the Carpathian Mountains.

Contour Lines

The data for the contour lines was downloaded from USGS – here SRTM (Shuttle Radar Topography Mission) data by the NASA is available. This is, in part of the world, available in 1 arc-second resolution (30m at the Equator), and in 3 arc-second resolution (90m at the Equator) for the whole world (below 60° latitude). Data for Romania can be found here.

The website provides raw DEM data in *.hgt format files that are compressed. The data is sectioned in tiles, according to latitude and longitude. For Romania 56 tiles were used: from N44E020 to N48E030.

After downloading, the data needs to be converted in the proper format. There is a popular script online that converts the SRTM HGT format into .tif format. It's basically a georeferenced image file with elevation data represented as different brightness values. The script can be found here (a preview and explanation are shown below. This is an adapted script for this project).

The bash file creates two of formats: *bil, and *.tif (GEOTIFF). The image format (tif) is the only one needed in the following steps. In order for the contour lines to appear on the map, they need to be on the server, but the database does not support raster images, so they need to be converted in a format that can be stored by PostgreSQL. Another option is to transform the tif image into shapefiles and store them in a folder on the computer, then modify the stylesheet for each set of contour lines and at the data source section (in the XML file) type the hardcoded location of the contour lines. We tried both possibilities, but without success so far.

First we need to make sure the reference system is specified in the file (which is not always the case):

The image is warped in the next step in the Spherical Mercator Projection, because this is what Mapnik actually reads:

Possibility 1: Import the DEM data into the PostGis database

For this step the gdal tool was used, which translates the image into a shapefile:

For <step>, the elevation difference between adjacent contour lines, in the units used by the DEM was specified. In this case, this is meters. The -snodata option specifies the value used in the image tiles to denote missing elevation values (which is -32768 for current CIAT/CSI data). Now, the contour lines can be imported into the PostGIS database:

Shp2pgsql is a tool that imports the shapefiles into the database. In this case a bash file was ran because of the big amount of data that we had. Every set of contour lines was imported in the database, using this script. The next step is to add styles for the contour lines in the stylesheet, so that Mapnik can read them and when rendering the tiles, they will be displayed on the map.

Style adaptation for Mapnik

In the stylesheet, it is required to add the contour lines layers (56 in this case), where one wants them to appear - in this case after the landcover layers, so they will appear on top of them. The layers contain the style rules and the data source (where these layers are stored, where can the rendering program can find them). The first section that needs to be added is the style section, where the style rules are defined (see below):

The Maximum Denominator value represents the zoom level at which the contours can be seen. In the example above, the contours will be seen at zoom level 4 for the first time. We have tried different zoom levels, but the result is the same – the contour lines do not appear on the map. The Line symbolizer element defines the color and the width of every line. These values can be modified every time, it depends on the user’s preferences.

After the style rules are added to the XML file, the layer elements should also be added (an example is provided in the figure below):

Every layer contains the style rules defined earlier and the source of the data (the gis database).

Since this method was not successful, we tried also the shapefile approach.

Possibility 2: The shapefile approach

This method requires that the shapefiles remain on the computer (which acts like a server) and in the stylesheet the data source (the path where the files are stored) should be linked to each layer defined in the XML stylesheet, like shown below:

The path for the layer above is a folder that contains all the shapefiles. The layer above is one of the 10 m contours (we generated 10 m, 50 m and 100 m contour lines for each set of DEM).

This method was also not successful, even when changing the projection or adding the contour layers and styles as the first elements that should appear on the map. It might be to be a problem with the arrangement in the XML stylesheet, that was not noticed, even though more than enough time was spend dealing with this problem. This method works very good for the hillshading layers, but not for the contour lines.

Hillshading

The hillshading of the map was done from the georeferenced and warped images used in the contour lines step. It is a simple command from the Ubuntu’S terminal:

One of the resulting shaded images is presented below:

Like we did for the contour lines, we also need to define a style for the “hillshade layers” and add these layers to the XML stylesheet. Because these elements are raster formats, it is a difference in defining the styles and the layers have other properties than the contour lines (see figure below):

These layers were added as first layers and the following layers (just the ones that are actually polygons, like landcover elements) were set to a 50% opacity, so that the 3D effect is more visible. (the attribute for that is fill-opacity=”0.5”).

The rules for raster formats include the opacity (in this case it is set to 100%, so that the 3D effect will be more obvious) and the scaling which can be bilinear, fast and bilinear8. These Three options, defined by the XML Schema for Mapnik, define the interpolation method: bilinear uses the bilinear interpolation for all 4 channels (RGBA), bilinear8 assumes just one channel and fast uses the nearest neighbor. In this case the bilinear attribute was implemented.

The georeferenced files are stored on the computer, so the data source is the path where they are located. For a raster layer, the type and the format also need to be defined: they have the type set to gdal because this was the method used to create them and the format is tiff, of course.

**Note: It is recommended to run the configuration file of renderd after every modification of the stylesheet, so that eventual errors (most of the time they come from mistyping) can be detected. A performant text editor is also recommended. We used in thjis case Atom Editor (also open source).

After starting the rendering process and while the tiles are loaded at different zoom levels, the hillshade effect appears on the website: