wiki:CowsFramework/CowsInstallation/CowsPython2.6

Version 5 (modified by astephen, 9 years ago) (diff)

--

Installing the COWS stack in Python 2.6

Introduction

These notes were initially developed as part of the process of porting the COWS stack (== cows + cowsserver + cowsclient) to Python 2.6 and Pylons 1.0 (see the migration to python 2.6 page. They also serve as a useful set of guidelines on how to build the COWS stack including a number of dependencies that require some manual intervention in the build process.

COWS server and COWS client

Since the COWS server dependencies are a superset of the COWS client dependencies you the installation procedure for the COWS server is presented in full and you can infer the subset of requirements for the COWS client.

Installing COWS server dependencies

The COWS Server compatible with Pylons 1.0 and Python 2.6 has the following dependencies. This list was produced using the pip freeze tool:

Beaker==1.5.4
FormEncode==1.2.3dev
Genshi==0.6
Mako==0.3.4
MarkupSafe==0.11
OWSLib==0.3.1
PIL==1.1.7
Paste==1.7.5.1
PasteDeploy==1.3.4
PasteScript==1.7.3
Pylons==1.0
Routes==1.12.3
Shapely==1.2.5
Tempita==0.5dev
WebError==0.10.2
WebHelpers==1.2
WebOb==1.0
WebTest==1.2.2
basemap==1.0
cdat-lite==5.2-1
-e svn+http://astephen@proj.badc.rl.ac.uk/svn/ndg/cows/branches/migrate-py26-pylons10@7492#egg=cows-1.5.1_r7492-py2.6-migrate-py26-pylons10-r7492
-e svn+http://astephen@proj.badc.rl.ac.uk/svn/ndg/cowsclient/branches/migrate-py26-pylons10@7502#egg=cowsclient-1.5.1_r7502-py2.6-migrate-py26-pylons10-r7502
-e svn+http://astephen@proj.badc.rl.ac.uk/svn/ndg/cowsserver/branches/migrate-py26-pylons10@7490#egg=cowsserver-1.5.1_r7490-py2.6-migrate-py26-pylons10-r7490
csml==2.7.13
decorator==3.2.0
-e svn+http://astephen@proj.badc.rl.ac.uk/svn/ndg/qesdi/geoplot/trunk@6519#egg=geoplot-0.3.3_r6540-py2.6-dev_r6519
-e svn+http://astephen@proj.badc.rl.ac.uk/svn/ndg/qesdi/image_utils/trunk@5700#egg=image_utils-1.0_r5700-py2.6-dev_r5700
libxml2-python==2.6.21
libxml2dom==0.4.7
matplotlib==1.0.0
mock==0.7.0b2
nose==0.11.4
numpy==1.3.0
pycairo==1.2.2
simplejson==2.1.1
wsgiref==0.1.2

The rest of this page explains how to

COWS Server (and COWS and COWS client) installation (with dependencies)

These instructions suggest installing the COWS server into a  virtualenv self-contained python environment. They also suggest the use of  pip as the installation tool. Alternatively, buildout can be used.

Note that not everything builds automatically and there are various stages when environment variables need to be set to pick up appropriate versions of packages.

I have tried to order the dependency list correctly to avoid having to re-build any packages (hopefully!).

1. Non-python packages

NetCDF

Version: 3.6.3

Manually install NetCDF libraries (Fortran not needed):

$ cd $HOME
$ mkdir -p external/netcdf/src
$ cd external/netcdf/src
$ wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-3.6.3.tar.gz
$ tar xvzf netcdf-3.6.3.tar.gz
$ cd netcdf-3.6.3/
$ ./configure --enable-shared --disable-f90 --disable-f77 --prefix=$HOME/external/netcdf
$ make
$ make install
$ make check

Cairo

Version: 1.2.4

Probably exists on any server, but can be built by Yast or package manager.

GEOS

GEOS is needed for libgeos_c.so file for python, used in matplotlib, basemap and shapely.

$ cd
$ cd external/netcdf/src
$ wget http://download.osgeo.org/geos/geos-3.2.2.tar.bz2
$ bunzip2 geos-3.2.2.tar.bz2
$ tar xvf geos-3.2.2.tar
$ cd geos-3.2.2
$ mkdir $HOME/external/geos
$ ./configure --prefix=$HOME/external/geos
$ make
$ make install

2. Python packages

First activate the virtualenv environment:

$ cd $HOME
$ . ENV/bin/activate

Most of the following packages was installed by typing:

$ cd $HOME
$ pip install <package>==<version>

# E.g.

$ pip install numpy=1.3.0

Now start installing...

Numpy

Version: 1.3.0

pip installed egg: numpy-1.3.0-py2.6-linux-x86_64.egg

pip install numpy==1.3.0

CdatLite

Version: 5.2_1

pip installed egg: cdat_lite-5.2_1-py2.6-linux-x86_64.egg

$ export NETCDF_HOME=$HOME/external/netcdf
$ pip install cdat_lite==5.2_1

CSML

Version: 2.7.13

easy_install installed egg: csml-2.7.13-py2.6.egg

easy_install --install-dir=ENV/lib/python2.6/site-packages http://ndg.nerc.ac.uk/dist/csml-2.7.13-py2.6.egg

Pycairo

Version: 1.2.2

Needed to manually install this to use compatible version with system cairo version.

Needed to build pycairo compatible with release on westerly of cairo (1.2.4) so used 1.2.2 of pycairo. Needed to download manually rather than use pip (because too old?).

$ cd $HOME/external/src
$ wget http://cairographics.org/releases/pycairo-1.2.2.tar.gz
$ tar xvzf pycairo-1.2.2.tar.gz
$ cd pycairo-1.2.2/
$ # Need to make this dir for it to install OK
$ mkdir $HOME/python26/lib/pkgconfig
$ python setup.py build
$ python setup.py install

Test with:

$ python -c 'import cairo'

OLD: Tested with (a line stolen from geoplot): OLD:  http://proj.badc.rl.ac.uk/ndg/browser/qesdi/geoplot/trunk/lib/geoplot/custom_cairo_renderer.py#L9):

matplotlib

Version: 1.0.0

Using egg: matplotlib-1.0.0-py2.6-linux-x86_64.egg

Cannot install with PIP. Need to extract the egg and build locally, as instructed on wiki page at:

 http://proj.badc.rl.ac.uk/dcip/wiki/UkcipDdp/Deployment/MatplotlibInstall

$ # We'll extract an editable version of matplotlib to ./matplotlib
$ cd $HOME/external/src
$ easy_install -eb . matplotlib==1.0.0
$ # Note the "pip install --src=. matplotlib==1.0.0" should also work
$ cd matplotlib
$ # Create a setup config file
$ cp setup.cfg.template setup.cfg

Now edit the config file so it contains the following lines:

#tag_svn_revision = 1
---
backend = Cairo
---
#
# Added datapath to point to mpl-data dir
datapath=/disks/westerly1/bull/ENV/etc/mpl-data

Make the ~/ENV/etc dir to put the matplotlib data files in:

$ mkdir ~/ENV/etc

Now create the egg:

$ python setupegg.py bdist_egg

Now copy the new egg to site-packages:

$ cp dist/*.egg $HOME/ENV/lib/python2.6/site-packages/
$ export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/matplotlib-1.0.0-py2.6-linux-x86_64.egg

And copy the data files into the etc dir:

cp -r lib/matplotlib/mpl-data $HOME/ENV/etc/

And set data path:

export MATPLOTLIBDATA=$HOME/ENV/etc/mpl-data

Test with:

$ python -c 'import matplotlib'
$ python -c 'from matplotlib.backends.backend_cairo import RendererCairo, FigureCanvasCairo, cairo, Affine2D'

Basemap

Version: 1.0

Using egg: basemap-1.0-py2.6-linux-x86_64.egg

Need to extract the egg and build locally, as instructed on wiki page at:

 http://proj.badc.rl.ac.uk/dcip/wiki/UkcipDdp/Deployment/MatplotlibInstall

$ # Environment needs some of these...
$ export GEOS_DIR=$HOME/external/geos
$ export LD_LIBRARY_PATH=$HOME/external/geos/lib

OLD*:$HOME/external/geos/lib/libgeos_c.so

$ # We'll extract an editable version of basemap to ./basemap
$ cd $HOME/external/src
$ easy_install -eb . basemap
$ cd basemap
$ python setupegg.py bdist_egg

Now copy the new egg to site-packages and setup basemap-data:

$ cp dist/*.egg $HOME/ENV/lib/python2.6/site-packages/
$ cp -r lib/mpl_toolkits/basemap/data $HOME/ENV/etc/basemap-data
$ export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/basemap-1.0-py2.6-linux-x86_64.egg
$ export BASEMAPDATA=$HOME/ENV/etc/basemap-data

Environment seemed to then need: $ export DBFLIBSO=$HOME/svn/cowsserver/buildout/basemap/build/lib.linux-x86_64-2.6

Test with:

$ python -c 'import pylab;from mpl_toolkits.basemap import Basemap; b = Basemap();b.drawcoastlines(); pylab.savefig("/tmp/map.png")'
$ python -c 'from mpl_toolkits import basemap; x =basemap.Basemap(llcrnrlon=-10.5,llcrnrlat=49.5,urcrnrlon=3.5,urcrnrlat=59.5,resolution="i",projection="tmerc",lon_0=-4.36,lat_0=54.7,rsphere=[6370997,1])'

PIL

Version: 1.1.7

pip installed egg: PIL-1.1.7-py2.6-linux-x86_64.egg

$ pip install pil==1.1.7

Test with:

$ python -c 'import Image'

Geoplot

Version: taken from trunk and then edited as follows:

$ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/qesdi/geoplot/trunk geoplot
$ cd geoplot

Now edit the following line:

$ svn diff lib/geoplot/grid_builder_national.py
                                urcrnrlon=10, urcrnrlat=10,
+                               rsphere=[6370997,1],
                                resolution=None)

TOFIX: This fix needs checking.

Test with:

$ python -c 'import geoplot.grid_builder_national'

image_utils

Version: 1.0

Installed from SVN repository:

$ cd $HOME/external
$ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/qesdi/image_utils/trunk image_utils                     
$ python setup.py develop

Test with:

$ python -c 'import image_utils'

Pylons

Version: 1.0

When you pip install pylons-1.0 you get the following dependencies installed as eggs:

Beaker-1.5.4-py2.6.egg
decorator-3.2.0-py2.6.egg
FormEncode-1.2.3dev-py2.6.egg
Mako-0.3.4-py2.6.egg
MarkupSafe-0.11-py2.6-linux-x86_64.egg
nose-0.11.4-py2.6.egg
Paste-1.7.4-py2.6.egg
PasteDeploy-1.3.4-py2.6.egg
PasteScript-1.7.3-py2.6.egg
Routes-1.12.3-py2.6.egg
simplejson simplejson-2.1.1-py2.6-linux-x86_64.egg
Tempita-0.5dev-py2.6.egg
WebError-0.10.2-py2.6.egg
WebHelpers-1.2-py2.6.egg
WebOb-1.0-py2.6.egg
WebTest-1.2.2-py2.6.egg

So try it:

$ pip install pylons==1.0

Mock

Version: 0.7.0b2

Pip installed egg: mock-0.7.0b2-py2.6.egg

$ pip install mock==0.7.0b2

Shapely

Version: 1.2.5

Pip installed egg: Shapely-1.2.5-py2.6.egg

$ pip install shapely==1.2.5

Genshi

Version: 0.6

Pip installed egg: Genshi-0.6-py2.6.egg

$ pip install genshi==0.6

owslib

Version: 0.3.1

pip installed egg: OWSLib-0.3.1-py2.6.egg

$ pip install owslib==0.3.1

cows cowsserver

3. Python packages only used for testing

libxml2

Version: 2.6.21

Need to install lib2xmldom which requires libxml2. Need to download source as no egg available.

OLD: NOT NEEDED: lxml-2.3beta1-py2.6-linux-x86_64.egg

$ cd $HOME/external
$ wget ftp://xmlsoft.org/libxml2/python/libxml2-python-2.6.21.tar.gz
$ tar xzvf libxml2-python-2.6.21.tar.gz
$ cd libxml2-python-2.6.21/
$ python setup.py build
$ python setup.py install

Test with:

$ python -c 'import libxml2'

libxml2dom

Version: 0.4.7

pip installed egg: libxml2dom-0.4.7-py2.6.egg

pip install libxml2dom==0.4.7

Test with:

$ python -c 'import libxml2dom'

Installing cows libraries

At present I've installed from the migrated branch and used develop eggs.

COWS:

$ cd $HOME/external
$ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/cows/branches/migrate-py26-pylons10 cows
$ cd cows
$ python setup.py develop

COWS Server:

$ cd $HOME/external
$ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/cowsserver/branches/migrate-py26-pylons10 cowsserver
$ cd cowsserver
$ python setup.py develop

COWS Client:

$ cd $HOME/external
$ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/cowsclient/branches/migrate-py26-pylons10 cowsclient
$ cd cowsclient
$ python setup.py develop

Setting up the environment to run

The following additions were appended to the activate script in the virtualenv:

# Additional environment variables required
export GEOS_DIR=$HOME/external/geos
export LD_LIBRARY_PATH=$HOME/external/geos/lib
export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/matplotlib-1.0.0-py2.6-linux-x86_64.egg
export MATPLOTLIBDATA=$HOME/ENV/etc/mpl-data
export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/basemap-1.0-py2.6-linux-x86_64.egg
export BASEMAPDATA=$HOME/ENV/etc/basemap-data

Then the environment can be set up with:

$ . $HOME/ENV/bin/activate

Testing running a service in the above environment

To test the service, refer to the notes on the Getting Started with COWS page using the start directories for COWS server and COWS client apps as:

$ $HOME/external/cowsserver
$ $HOME/external/cowsclient