wiki:BuiltoutWalkthrough

Version 1 (modified by spascoe, 11 years ago) (diff)

--

Using zc.buildout to deploy Pylons apps

This is some notes on how to use  zc.buildout to control the eggs being used to deploy a pylons application. I've presented it in terms of a walkthrough of converting a real running BADC service, the NERC Portals Project MIDAS visualisation service.

This walkthrough also demonstrates how to configure a pylons application egg to automatically generate a default initiation file with the paster make-config command and how to migrate a pylons app from using ows_common-refactor to it's successor, COWS.

The Status Quo

We have found that running multiple services on one box can be problematic if the services require different versions of an egg. The classic example is superglue running several pylons apps for the NERC Portals project. some apps require the ows_common and ows_server from the trunk and some require ows_common from the refactor branch. Different versions of csml and cdms are required depending on whether we are using cdat-5 or cdat-4.

The solution we have been using is to have a deployment directory with it's own eggs directory. PYTHONPATH is set to the eggs directory before running paster. For instance superglue:/usr/local/csmlGrapher_deployment looks like this

$ ls
capabilities.conf                     csml_config/  paster.log  users.txt~
csml2kmlpylon-1.0.1b_r3819-py2.5.egg  deploy.ini    paster.pid
csml2kmlpylon.conf.xml                deploy.ini~   pilfonts/
csml2kmlpylon.conf.xml~               eggs/         users.txt
-bash-3.00$ ls eggs
cddump                                 csmlscan
cdscan                                 easy-install.pth
convertcdms                            easy-install.pth~
csml2kml-1.0b3-py2.5.egg               env.sh
csml2kmlpylon-1.0.1b-py2.5.egg/        env.sh~
csml2kmlpylon-1.0.1b_r3784-py2.5.egg/  ows_common-0.1.1dev_r3812-py2.5.egg
csml2kmlpylon-1.0.1b_r3818-py2.5.egg/  paster.log
csml2kmlpylon-1.0.1b_r3819-py2.5.egg/  site.py
csml2kmlpylon-1.0b-py2.5.egg/          site.pyc

The eggs directory contains various versions of the pylons app egg (csml2kmlpylon) and it's own version of ows_common and csml2kml. It's a system that works but is fiddly to setup.

There is a tool that works very well to build custom python egg environments called [ http://pypi.python.org/pypi/zc.buildout zc.buildout]. In this walkthrough we will convert this layout as follows:

  1. Replace ows_common-... with cows-0.2rc1
  2. Use zc.buildout to create a deployment directory similar to above automatically.

Adapting the Pylons App

A few changes to the csml2kmlpylon source are needed first.

We need csml2kmlpylon to use cows rather than the refactored ows_common. This requires trivial changes to the import statements. See [4012] for what I did.

Pylons provides help in creating ini files on deployment. See  http://wiki.pylonshq.com/display/pylonsdocs/Packaging+and+Deployment for details. In [4014] I edit the the deployment ini file template. Once this was done I could create a pylons ini file from the installed egg using

$ paster make-config csml2kmlpylon myconfig.ini

Finally I wanted to fix a problem where csml2kmlpylon expected static KML files to be inside the egg. This wasn't ideal as the files are generated elsewhere so I changed the app to allow an external static_files directory. See [4015].

Now ready to build the egg and put it on ndg.nerc.ac.uk/dist

$ python setup.py egg_info bdist_egg sdist
$ scp dist/csml2kmlpylon-1.0.2* badc@glue.badc.rl.ac.uk:/var/www/html/dist

Creating buildout.cfg

We are going to create a configuration file for use with the zc.buildout command buildout. With this configuration buildout will be able to download everything we need and create the deployment directory. For convenience I'll put buildout.cfg in the csml2kmlpylon svn tree but it doesn't go in the egg.

The config file was commited in [4016]:

[buildout]
parts: csmlgrapher

[csmlgrapher]
recipe: zc.recipe.egg
find-links: http://ndg.nerc.ac.uk/dist
# We include PasteScript get the paster command in our buildout directory
eggs: csml2kmlpylon
      cows>=0.2rc1
      csml2kml>=1.0b3
      PasteScript

Now upload buildout.cfg to ndg.nerc.ac.uk/dist so that we can download it where we want it.

$ scp buildout.cfg badc@glue.badc.rl.ac.uk:/var/www/html/dist/csml2kmlpylon-buildout.cfg

Installing zc.buildout

Now moving to superglue. We are going to use  zc.buildout to create a deployment of csml2kmlpylon with it's own copies of all the eggs it uses, including cows and cdms-lite. For this we need to install zc.buildout on superglue. It is actually possible to bootstrap zc.buildout without installing it system-wide, but this seemed unneccessary. We'll use easy_install:

superglue$ sudo easy_install zc.buildout

We now have the buildout command available.

Creating the deployment

This is where the magic comes. We simply download the buildout.cfg file and run it through buildout to create our deployment. We are going to create a new deployment directory (/usr/local/csml2kmlpylon_deployment) and replace /usr/local/csmlGrapher_deployment with it.

superglue$ mkdir /usr/local/csml2kmlpylon_deployment
superglye$ cd /usr/local/csml2kmlpylon_deployment
superglue$ wget http://ndg.nerc.ac.uk/dist/csml2kmlpylon-buildout.cfg
superglue$ buildout -c csml2kmlpylon-buildout.cfg

This will put all the eggs we need in ./eggs and put paster in ./bin

Configuring the Pylons app

Now we create the app config file using the mechanism we setup earlier. Note we use the buildout paster script.

superglue$ bin/paster make-config csml2kmlpylon deploy.ini

Look away now

At this point we hit reality. csml2kmlpylon isn't a perfect pylons app when it comes to configuration so I needed to do various things to configure it properly. If you are just interested in using buildout you can skip this section.

Some configuration and data files are needed that aren't created with paster make-config. This should be fixed with some refactoring and scripting but for the moment we'll just copy them from the old deployment.

superglue$ for f in csml_config pilfonts capabilities.conf csml2kmlpylon.conf.xml ; do cp -R ../csmlGrapher_deployment/$f . ; done
superglue$ mkdir kml
superglue$ find ../csmlGrapher_deployment -name \*.km? xargs cp $f kml

Also I noticed several app-specific things missing in deploy.ini that were needed. Thee AuthKit options were missing.

Swapping the deployments

We now swap the old deployment for the new one.

superglue$ pushd ../csmlGrapher_deployment
superglue$ paster serve deploy.ini stop
superglue$ pushd
superglue$ bin/paster serve deploy.ini --daemon

And the service is back up from the new deployment directory. Notice the second call to paster is the buildout paster script.

Finally we clearly label the old deployment as not in use. It is kept as a backup and as reference for this walkthrough:

superglue$ cat /usr/local/csmlGrapher_deployment/README
WARNING: This deployment is not in use.  It has been replaced by
    /usr/local/csml2kmlpylon_deployment.  It is retained here on a
    temporary basis for reference.

Stephen Pascoe.
superglue$ mv /usr/local/csmlGrapher_deployment /usr/local/csmlGrapher_depl_old