= Installing the Pylons stack: Discovery, Browse, and CSML services =

All the NDG services are now built on top of the the [http://pylonshq.com/ Pylons] framework. This is a guide to installing the framework and all the NDG services based on my experiences of installing from scratch on the glue development server. Most of the install uses 'easy_install' to install python eggs.

== Step 1: Choose your Python ==
This process is going to install a lot of packages (mainly eggs) to your python distrobution. You may or may not want to use the system python for this. However we are recommending python 2.4 for this install so if you system python is different you will want to create a separate python.

On Glue a fresh python 2.4 was installed into /usr/local.

Whichever python you use (system or otherwise), in the following instructions, whenever you see the command 'python' it refers to your chosen python so make sure your PATH environment variable is correct. The command 'sudo' is used throughout as I was in a directory owned by the root user, but this may or may not apply to you. 

Python 2.4 can be downloaded [http://www.python.org/download/releases/2.4.4/ here] along with install instructions.

== Step 2: Install the latest setuptools egg ==

Download ez_setup.py:
{{{
sudo wget http://peak.telecommunity.com/dist/ez_setup.py
}}}

If that didn't work you probably need to set the http_proxy environment variable:
{{{
export http_proxy=http://yourproxyurl.com:8080
}}}

Run it and setuptools will be installed:
{{{
sudo python ez_setup.py
}}}

And you can now delete ez_setup.py:
{{{
sudo rm ez_setup.py
}}}

== Step 3: Install Paste ==
{{{
sudo python -m easy_install paste
}}}

== Step 4: Install Pylons ==
{{{
sudo python -m easy_install pylons==0.9.5
}}}
If easy_install can't find this version, try:
{{{
sudo python -m easy_install -f http://pylonshq.com/download/0.9.5/ Pylons==0.9.5
}}}

== Step 5: Install Kid templating plugin ==
{{{
sudo python -m easy_install TurboKid
}}}

== Step 6: Install CSML ==
This stage will install the CSML, Cdat_lite and Numeric eggs. The Numeric install was not straightforward on Glue, perhaps because I wasn't using the system python.

Anyway, try this:
{{{
sudo python -m easy_install -f http://ndg.nerc.ac.uk/dist csml
}}}

If that worked go straight to Step 7.

However if you got  a message about not being able to find Numeric/arrayobject.h then then try and follow these instructions to make those headers available (there's probably a better way to do this but this is what worked for me). I think this is due to trying to install into a non-system python.

=== Step 6.1: Download and unpack the Numeric source code === 
{{{
wget http://downloads.sourceforge.net/numpy/Numeric-24.2.tar.gz
sudo gunzip  -c Numeric-24.2.tar.gz
sudo tar -xvf  Numeric-24.2.tar     

cd Numeric-24.2
}}}

=== Step 6.2: Copy header files into your local includes ===
I'd installed python into /usr/local, so I had to move the headers to /usr/local/include/python2.4

There is a subdirectory called Numeric which contains the header files, copy this into your python includes eg:
{{{
sudo cp -rf Numeric /usr/local/include/python2.4/
}}}

And there are some more that need moving:
{{{
sudo cp -rf Packages/FFT/Include /usr/local/include/python2.4
sudo cp -rf Packages/RNG/Include /usr/local/include/python2.4
}}}

Right, now you can get rid of this source directory.
{{{
cd ..
sudo rm -rf Numeric-24.2
}}}

=== Step 6.3: Install Numeric ===
{{{
sudo python -m easy_install http://ndg.nerc.ac.uk/dist/Numeric-24.2.tar.gz
}}}

=== Step 6.4: Install Cdat_Lite ===
{{{
sudo python -m  easy_install http://ndg.nerc.ac.uk/dist/cdat_lite-4.1.2-0.2.4.tar.gz
}}}

=== Step 6.5: Install CSML ===
{{{
sudo python -m easy_install -f http://ndg.nerc.ac.uk/dist csml
}}}

== Step 7: Recap ==
If you've got to this stage you should be able to start python and test out your imports:

{{{
-bash-2.05b$ python
Python 2.4.4 (#6, Jul 10 2007, 09:11:51) 
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pylons
>>> import paste
>>> import Numeric
>>> import cdms
>>> import csml
could not import Image
>>> 
}}}

The message about 'Image' means the python imaging library is not installed. This is not mandatory, and is only needed for PML at the moment.

So now all the components are there except for the discovery, browse and CSML services code...

== Step 8: Install the OWS Framework Eggs ==
Install `ows_server` egg:
{{{
sudo python -m  easy_install -f http://ndg.nerc.ac.uk/dist ows_server
}}}

Install `ows_common` egg:
{{{
sudo python -m  easy_install -f http://ndg.nerc.ac.uk/dist ows_common
}}}

== Step 9: Set-up Config directories ==
Make a directories for configuration and items:
{{{
sudo mkdir /etc/ndg
sudo mkdir /etc/ndg/ows_server
sudo mkdir /etc/ndg/ows_server/conf /etc/ndg/ows_server/run /etc/ndg/ows_server/logs
sudo mkdir /etc/ndg/ows_server/conf/certs 
}}}

Copy config files from the repository to `/etc/ndg/ows_server/conf`

{{{
sudo cd /etc/ndg/ows_server/conf
svn export svn+ssh://pjkersha@glue.badc.rl.ac.uk/ndgsvn/TI05-delivery/ows_framework/trunk/ows_server/development.ini svn export 
svn export svn+ssh://pjkersha@glue.badc.rl.ac.uk/ndgsvn/TI05-delivery/ows_framework/trunk/ows_server/ndgDiscovery.config ndgDiscovery.config 
}}}

== Step 10: Edit local settings ==
You will need to configure which port you are using to serve the pylons applications. The default is `8080`. To change it edit the file called `development.ini` (in the `/etc/ndg/ows_server/conf directory`) and change the port number in this section to the port you want to use (or leave it if you want 8080).

{{{
[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 8080
}}}

Now open the file in `/etc/ndg/ows_server/conf` called `ndgDiscovery.config`, and change the 'repository' and 'server' urls to match your server.

Now ensure you are in the NDG Services config directory:
{{{
cd /etc/ndg/ows_server/conf
}}}

Contact [mailto:P.J.Kershaw@rl.ac.uk Phil] in order to get help making the security settings under `[NDG_SECURITY]` section of the config file.

== Step 11: Start the services ==
Starting the server is a one line command:
{{{
paster serve --reload development.ini
}}}

If that all worked, then you should be able to navigate to your server eg http://localhost:8080 and see the message:
{{{
Welcome to your Pylons Web Application
}}}

If that worked, try navigating to the discovery service:
e.g.
{{{
http://localhost:8080/discovery
}}}

If that worked try and view discovery and browse records!!

CSML services (WMS and WCS) are also available in this stack, but as they are still under development and nobody has any CSML they will be the subject of another wiki page.

== Step 12: Add to boot ==
This script can be put in /etc/init.d and will start your service on boot. Give it a suitable name, such as 'ndgservices'.

{{{
#!/bin/sh -e

export PYTHON_EGG_CACHE=/tmp
PID=/etc/ndg/ows_server/run/paster.pid
INI=/etc/ndg/ows_server/conf/development.ini
LOG=/etc/ndg/ows_server/logs/paster.log
PROG=/usr/local/bin/paster

status()
{
    local pid=
    
    # Get pid from PID file
    local pidFound=
    if [ -f $PID ] ; then
        pid=$(cat $PID)
        if [ -z "$pid" ]; then
            echo $"Can't get pid from pid file $PID"
            return
        fi
        pidFound=Yes
    fi

    # look for pid in listing
    for i in `pidof -o $$ -o $PPID -o %PPID -x "${PROG}"`; do
        [[ $i = $pid ]] && pidFound=Yes && break;
    done
    
    if [ -n "$pidFound" ]; then
        echo $"$prog (pid $pid) is running..."

    elif [ -f ${PID} ]; then
        echo $"$prog is dead but pid file $PID exists"
    else
        echo $"$prog is dead"
    fi
}

case "$1" in
  start)
    ${PROG} serve --daemon --reload --pid-file=$PID --log-file=$LOG $INI
    ;;
  stop)
    ${PROG} serve --stop-daemon --pid-file=$PID
    ;;
  restart)
    ${PROG} serve --stop-daemon --pid-file=$PID
    ${PROG} serve --daemon --reload --pid-file=$PID --log-file=$LOG $INI
    ;;
  status)
    status
    ;;
  *)
    echo $"Usage: $(basename $0) {start|stop|restart|status}"
    exit 1
esac
}}}

you can then start/restart/stop/check status using:
{{{
sudo sh /etc/init.d/ndgservices start
sudo sh /etc/init.d/ndgservices restart
sudo sh /etc/init.d/ndgservices stop
sudo sh /etc/init.d/ndgservices status
}}}

(in this case 'ndgservices' is the name of the script)

= ISSUES LOG =

Please note any problems encountered or issues here - along with the solution if you know it.

== Issue 1. Numeric Install == 
The Numeric install doesn't always go smoothly, see step 6 for possible solution.

== Issue 2. Routes not working properly. ==
If you try a URL such as this:
{{{
http://localhost:8080/view/badc.nerc.ac.uk__NDG-B1__dataent_COAPEC
}}}

And get a "Not Found, The resource could not be found." error then that indicates a problem with the routes module. 

This problem occurs with routes version 1.6.3. Upgrading to 1.7 should solve the problem. Use the -U flag in easy_install to upgrade:
{{{
easy_install -U routes
}}}

== Issue  3. Zombies ==
When trying to set up step 12 (run at boot) I was getting lots of problems with zombie processes that I couldn't kill off. It turned out that the paster.pid file was not getting written out. If this happens edit the init script and change the location of the paster.pid file to somewhere writeable (or change the permissions of it's current location).