OSM Tile Creation on AWS Spot

This article describes some efforts to automate OSM tile creation on AWS spot instances.

I'm running a website for a regulars table in my city. As part of this regulars table we have a database of all pubs we visited, because we're in a different pub every Tuesday. Why not visualize this via OpenStreetMap? We wanted to create a nice slippy map with all pubs we had been to in the last 15 years. But OSM advises not to use their tile server, because they don't have much resources for the infrastructure available.

So the logical thing was to create a custom tileset on AWS spot and put it on S3 for serving to our website's visitors. This describes the automation that went into spinning up a spot instance and downloading the generated tiles to my local machine. AWS spot is a "spot market" for AWS instances that offers deep discounts on infrastructure. However if you host on AWS spot, your machines might go away at any time without notice. Spot is dynamically priced, so we're essentially one bidder among many.

The code

The code is available on Github at nomadenorg/osmtilegen. There is a set of scripts:

gen-tiles.py is our custom Python script to generate the tiles. This limits tile generation to the Hamburg (our city) bounding box.
mapgen.sh, this runs on the instance, it installs carto and mapnik, as well as postgresql version 10 and some utilities.
request.py is a boto3 script that does a spot request and prints the public hostname to stdout
spot-spec.json is the spot instance launch specification, this enables you to customize what instance config you'll end up with.
run.sh is the main driver that integrates it all.

When you start up run.sh it will run for approximately 30 minutes to generate the file tiles.tar, which will be downloaded to the local directory. In out case this is around 250 MB in size. The cost for a complete tile set is around $0.50 if I run in eu-west-1.

Extension points

There are some points where you can customize this workflow.

First you will want to roll your own gen-tiles.py to adapt the bounding box to your needs. Then you will also have to change the extract downloaded from the Hamburg extract to something else, maybe even the complete planet.

After you've done this you will probably want to start off with the OpenStreetMap Carto theme and customize it to your liking.

Both adaptions are rather easily done with the current setup.

Image credit: By GT1976 [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)], from Wikimedia Commons Wiki Commons