Importing boundaries from GADM.org
GADM is a database of administrative areas from around the world. If you are setting up a MapIt for your country then it is likely that this is a good starting point. At mySociety it is what we used for Kenya and Nigeria.
Note that the GADM data is freely available for academic and other non-commercial use. Redistribution, or commercial use, is not allowed without prior permission from them.
These are step by step instructions for loading all the areas for a country (we’ll use Nigeria) into MapIt:
Find and download the boundaries
- Go to the download page: http://www.gadm.org/country
- Select the country (in our case Nigeria)
- Select the “Google Earth .kmz” format
- Click “OK” to show the available downloads
- Download all the “levels” to your computer – for Nigeria there are levels 0, 1 and 2.
[ TODO: Talk about shapefile downloads here too – some have more data than KML. ]
Preview the downloaded files
Once the data files have been downloaded you can use the free Google Earth software to view the boundaries. It is available for download from: http://www.google.com/earth/explore/products/desktop.html
Prepare the files for import
- Collect all your files in one folder - we’ll use
Convert the files to kml (kmz is just kml gzipped):
cd /tmp/gadm_data # decompress all the kmz files unzip '*.kmz'
Work out the area types you’ll need
There will be several different types of area you’ll want to store. For example there will be the country boundary, and there may also be states or counties. Look in the kml files to find the different areas represented there.
This command will list them all:
grep '<description>' *.kml | sort | uniq -c | grep -v 'href='
For the Nigerian kml files we get this output:
1 NGA_adm0.kml:<description><![CDATA[NGA]]></description> 37 NGA_adm1.kml:<description><![CDATA[State]]></description> 1 NGA_adm1.kml:<description><![CDATA[Water body]]></description> 775 NGA_adm2.kml:<description><![CDATA[Local Authority]]></descri...
The bit we’re looking for is in the
CDATA[[...]] block. The number at the
start of the line in the number of occurrences.
Note that there is a single ‘Water Body’ in one of the kml files - using a text
editor we’ll remove that manually as it is not relevant to us. This is done by
finding the ‘Water body’ text and then deleting everything from and including
<Placemark> before it and up to and including the closing
Alternatively you could import using the steps below and then find it in the admin interface and delete it.
Prepare the MapIt install for the import
We’ll assume that you are using a locally running instance of MapIt. If not please change the URLs to match your setup.
- Install MapIt as per the installation instructions.
- Start the dev server.
- We’re assuming that your database is empty. If this is not the case you may have some conflicts.
Either visit the admin at http://127.0.0.1:8000/admin and add a generation there, with description ‘GADM version 2.0’, or run the following (which does the same thing):
./manage.py mapit_generation_create \ --desc='GADM version 2.0' --commit
- (optional) You can add the various Types in the admin interface now, or be prompted for them when you run the import script. If you want to use the admin interface, we’ll want a country - Nigeria with code N; a name type - ‘gadm’ with description ‘GADM English Name’; and appropriate area types, CTR “Country”, STA “State”, and LGA “Local Government Area”.
Import the kml
There is an import script that will look at the kml file and create entries from it. We’ll call it as follows:
./manage.py mapit_import \ --country_code N \ --generation_id 1 \ --area_type_code CTR \ --name_type_code gadm \ --commit \ /tmp/gadm_data/NGA_adm0.kml
You will be prompted to provide descriptions for the codes if you haven’t created them; we can use “GADM English Name” for gadm, “Country” for CTR, “State” for STA, “Local Government Area” for LGA, and “Nigeria” for ‘N’.
Repeat the above line for all your data files changing the path to the file and
--area_type_code to suit your import. You may also need to use a
different value for
--generation_id if this is not a fresh MapIt install.
If you want to try the import without committing to the database don’t specify
(If you get the error
django.db.utils.DatabaseError: invalid byte sequence for
encoding "UTF8": 0x00 you are probably being bitten by this
bug - the solution is to add
standard_conforming_strings = off to your
postgresql.conf file, or to apply
the patch in that ticket.)
Activate the generation
Once you are happy that the data is correct activate the generation in the admin interface, or run:
./manage.py mapit_generation_activate --commit
The same import script can import shapefiles too. You might need the extra
command line parameter of
--encoding if the encoding of the shapefile is not
UTF-8 (GADM is sometimes ISO-8859-1, for example).
You will also need to know which field in the shapefile contains the name of
the area, and specify it with the
--name_field parameter. If you run without
this parameter and ‘Name’ doesn’t work, the program will output a list of
possible choices that it could be.