wiki:UkcipDdp/Deployment/MatplotlibInstall

This page documents how to build eggs for matplotlib and basemap for later assembly into the ddp-trunk buildout directory. The instructions were written with generic MPL installations in mind rather than specifically the DDP deployment. DDP-specific notes are presented as a preface.

Variation on MPL notes for DDP

See notes inline on unusual install location of python packages in the case of SuSE. /usr/local/bin isn't on root's path so you often need to give the full path to easy_install and buildout (e.g. /usr/local/bin/easy_install). /usr/local/lib64/python2.5/site-packages must exist before installing setuptools. It doesn't unless you've installed some python packages first.

GEOS

geos-2.2.3 was installed on /common/packages/geos-2.2.3 so references to /usr/local/geos-2.2.3 can be replaced with /common/packages/geos-2.2.3. To run mpl make sure /common/packages/geos-2.2.3/lib is wired in to ld either with ldconfig or LD_LIBRARY_PATH.

buildout

There is a sample buildout config in /common/eggs/mpl_buildout.cfg. Either copy this and edit it or use the buildout:extends option to inherit the options there.

Building Matplotlib + Basemap eggs

For the UKCIP08 DDP project we are using Python Eggs to manage our software components where possible. To help us deploy multiple versions of the DDP application we are using the powerful tool zc.buildout to manage collections of eggs. This allows us to switch very easily between different configurations.

However, as always in the world of package management, things are not all plain saling when dealing with all our Python dependencies. In particular the plotting package Matplotlib and it's geographic extension Basemap don't work seemlessly with either easy_install or buildout. Here are some of the issues we have discovered.

  1. Both matplotlib and basemap have a separate setup script for building as an egg (setupegg.py). The only difference is that setupegg.py knows how to declare namespace packages. This design decision means easy_install (and buildout) cannot automatically build eggs for these packages from their tarballs.
  2. Both matplotlib and basemap require numpy to be present during execution of setup.py. This prevents buildout from building the eggs unless numpy is installed OUTSIDE THE BUILDOUT. It is partially a deficiency of buildout and partly because mpl/basemap don't use setuptools' setup_requires feature.
  3. Basemap requires geos-2.2.3. This is not the latest version of geos and isn't the version packaged for SuSE 10. It is bundled with basemap and can be configure/make/installed locally before running setup.py.

Dispite these problems it is possible to hand-build eggs for matplotlib and basemap that can then be assembled using buildout.

However, this approach violates another principle of good egg management. Namely that packages with C extensions depending on external libraries are best installed from tarballs. The method detailed below will create eggs for numpy, matplotlib and basemap labelled with the platform on which they were build but these eggs will not record their external C dependencies. For Matplotlib these are quite substantial as it relies on a large and configurable set of libraries, e.g. freetype, libpng and pygtk. The process below builds a minimal matplotlib to mitigate this problem. However the eggs generated need to be managed with care as they will break on other platform variants.

Aims

This method aims to install the latest matplotlib and basemap packages in a local environment (i.e. not in python's site-packages) using zc.buildout. We aim to install the minumum of dependencies system-wide.

Prerequesites

  1. Python 2.5.
  2. distutils. If the distutils module isn't importable from python try adding the python-devel package via your package manager.
  3. Connection to the web (i.e. HTTP proxy set if necessary).

Installing setuptools

Note about Suse10 on 64bit: SuSE has a strange python install system. If you haven't installed any python packages you may need to do the following before proceeding:

$ sudo mkdir -p /usr/local/lib64/python2.5/site-packages

Download the ez_setup.py script and run it as root:

$ wget http://peak.telecommunity.com/dist/ez_setup.py
$ sudo python ez_setup.py

Install zc.buildout (may need to use full path to easy_install on SuSE (/usr/local/bin/easy_install):

$ sudo easy_install zc.buildout

Installing numpy

Numpy must be installed first before the other eggs can be built. Although we could build it locally using virtual-env or easy_install wizardry we decided to install it system-wide as an egg.

To minimise the operations performed as root we download/build the egg first before installing it in site-packages:

# Build egg in current directory
$ easy_install -zmaxd . numpy
# Install build egg
$ sudo easy_install ./numpy*.egg

All being well numpy will download and compile. It will warn you that LAPAC isn't optimised but for data management tasks this isn't a problem. Test that we have it available:

% python
>>> import numpy
>>> numpy
<module 'numpy' from '/usr/local/lib64/python2.5/site-packages/numpy-1.2.1-py2.5-linux-x86_64.egg/numpy/__init__.pyc'>

This indicates you are using the numpy package version 1.2.1 from within an egg.

Building the matplotlib egg

Next we will build matplotlib as an egg. Although easy_install can't auto-build this from a tarball we can use it to grab the latest version:

$ easy_install -eb . matplotlib
$ cd matplotlib

We are now in the matplotlib distribution directory. Next we configure the setup procedure:

$ cp setup.cfg.template setup.cfg

Edit setup.cfg to your tastes. Comment out tag_svn_revision as it will incorrectly label your egg. For server applications you can switch off all the GUI backends by uncommenting the options in the [gui_support] section. This will minimise the number of dependencies you need.

Now create the egg. Note the custom setup script:

$ python setupegg.py bdist_egg

Your egg will be in the dist directory and should be moved to a common staging area. Assuming this is $MYEGGS:

$ cp dist/*.egg $MYEGGS

Building the basemap egg

WARNING: This method will build an egg that only works if geos-2.2.3 is installed and it's libraries available to ld.

Follow a similar process for basemap:

$ easy_install -eb . basemap
$ cd basemap

We need to build geos-2.2.3. In this case we will put it in /usr/local/geos-2.2.3 but provided GEOS_DIR is set correctly the compilation will work:

$ cd geos-2.2.3
$ ./configure --prefix=/usr/local/geos-2.2.3
$ make
$ sudo make install
$ export GEOS_DIR=/usr/local/geos-2.2.3

Build the egg and copy it to the staging area:

$ python setupegg.py bdist_egg
$ cp dist/*.egg $MYEGGS

Assembling a buildout

Describing the features of buildout is beyond the scope of this document but to create a python interpreter that sees these eggs do the following:

$ mkdir mpl_builout
$ cd mpl_buildout
$ buildout init

Edit buildout.cfg as follows, substituting <<MYEGGS>> for the path to the egg staging area. If the latest versions of matplotlib and basemap are not those in $MYEGGS you will need to declare the versions explicitly or buildout will try to get them from the internet:

[buildout]
parts = mpl

[mpl]
recipe = zc.recipe.egg
interpreter = python
find-links = /usr/local/src
eggs =
  matplotlib
  basemap

Make sure geos-2.2.3 is available to ld. In this case we will set LD_LIBRARY_PATH but on deployment it is better to use an /etc/ld.conf.d file. Then run buildout:

$ export LD_LIBRARY_PATH=/usr/local/geos-2.2.3/lib
$ buildout

You will have an interpreter in mpl_buildout/bin/python that you can test as follows:

% bin/python
>>> import pylab
>>> from mpl_toolkits.basemap import Basemap
>>> b = Basemap()
>>> b.drawcoastlines()
>>> pylab.savefig('map.png')

Look at map.png to check it contains a global map.