wiki:InstallDiscoveryBrowse

Version 34 (modified by pjkersha, 11 years ago) (diff)

--

Installing the Pylons stack: Discovery, Browse, and CSML services

All the NDG services are now built on top of the the  Pylons framework. This is a guide to installing the framework and all the NDG services based on my experiences of installing from scratch on the glue development server. Most of the install uses 'easy_install' to install python eggs.

Step 1: Choose your Python

This process is going to install a lot of packages (mainly eggs) to your python distribution. You may or may not want to use the system python for this. However we are recommending python 2.5 for this install so if your system python is different you will want to create a separate python.

On Glue a fresh python 2.5 was installed into /usr/local.

Whichever python you use (system or otherwise), in the following instructions, whenever you see the command 'python' it refers to your chosen python so make sure your PATH environment variable is correct. The command 'sudo' is used throughout as I was in a directory owned by the root user, but this may or may not apply to you.

Python 2.5 can be downloaded  here along with install instructions.

Step 2: Install the latest setuptools egg

Download ez_setup.py:

sudo wget http://peak.telecommunity.com/dist/ez_setup.py

If that didn't work you probably need to set the http_proxy environment variable:

export http_proxy=http://yourproxyurl.com:8080

Run it and setuptools will be installed:

sudo python ez_setup.py

And you can now delete ez_setup.py:

sudo rm ez_setup.py

Step 4: Install Pylons

sudo python -m easy_install "pylons>=0.9.6"

Step 5: Install Kid templating plugin

sudo python -m easy_install TurboKid

(Make sure you have at least 1.03)

Step 6: Install CSML

This stage will install the CSML, cdat_lite and numpy eggs. As of CDatLite 5.0, the Numeric package is replaced by numpy. These instructions are given here but more details of the new CDatLite package can be found here.

Download the netCDF tarball from  here. Extract and follow the instructions in the INSTALL file. If you encounter problems there is a  help page with a list of known issues

Install numpy:

sudo python -m easy_install numpy

Install CDatLite:

sudo python -m easy_install -f http://ndg.nerc.ac.uk/dist cdat_lite

Install CSML:

sudo python -m easy_install -f http://ndg.nerc.ac.uk/dist csml

Step 7: Recap

If you've got to this stage you should be able to start python and test out your imports:

-bash-2.05b$ python
Python 2.5.2 (r252:60911, Jun  4 2008, 11:30:27)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-59)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pylons
>>> import paste
>>> import numpy
>>> import cdms2
>>> import csml
NASAAmes interface not available. CSML will still work, but won't support NASA Ames files
Python Image Library not available. CSML will still work, but won't support image files
>>> 

The message about 'Image' means the python imaging library is not installed. This is not mandatory, and is only needed for PML at the moment.

So now all the components are there except for the discovery, browse and security code...

Step 8: Install the OWS Framework Eggs

Install ows_server egg:

sudo python -m  easy_install -f http://ndg.nerc.ac.uk/dist ows_server

Install ows_common egg:

sudo python -m  easy_install -f http://ndg.nerc.ac.uk/dist ows_common

Step 9: Install Security

For installation instructions for NDG Security Services see the  NDG Security Installation Guide. If security services are already configured but are running on a separate machine, install the security client software:

Download the special installation script:

wget http://ndg.nerc.ac.uk/dist/ndg-security-install.py

Run:

python ./ndg-security-install.py -a

For troubleshooting see the  Security installation Guide.

Step 10: Set-up Config directories

Make a directories for configuration and items:

sudo mkdir /etc/ndg
sudo mkdir /etc/ndg/ows_server
sudo mkdir /etc/ndg/ows_server/conf /etc/ndg/ows_server/run /etc/ndg/ows_server/logs
sudo mkdir /etc/ndg/ows_server/conf/certs 
sudo chmod 700 /etc/ndg/ows_server/conf/certs

The last step is an additional precaution to protect files in certs/

Copy config files from the repository to /etc/ndg/ows_server/conf

sudo cd /etc/ndg/ows_server/conf
svn export http://glue.badc.rl.ac.uk/ndgsvn/TI05-delivery/ows_framework/trunk/ows_server/development.ini 
svn export http://glue.badc.rl.ac.uk/ndgsvn/TI05-delivery/ows_framework/trunk/ows_server/ndgDiscovery.config

Step 10: Edit local settings

You will need to configure which port you are using to serve the pylons applications. The default is 8080. To change it edit the file called development.ini (in the /etc/ndg/ows_server/conf directory) and change the port number in this section to the port you want to use (or leave it if you want 8080).

[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 8080

Now open the file in /etc/ndg/ows_server/conf called ndgDiscovery.config, and change the 'repository' and 'server' urls to match your server.

Now ensure you are in the NDG Services config directory:

cd /etc/ndg/ows_server/conf

Discovery Standalone Interface

Note that it's possible to run the Discovery/Browse? code stack as a Discovery only interface. In this case, no security is required. To set in this configuration set the standalone flag in ndgDiscovery.config to 'True'.

Security Configuration

If Browse is run, then security needs to be configured. There are two components that need to be configured:

  1. Single Sign On Service - this enables uses from other trusted sites to login at your Browse site with their home ID.
  2. Gatekeeper - MOLES and CSML records available from the Browse interface can be secured with role based access control using the Gatekeeper.

Security settings are organised under the [NDG_SECURITY.*] sections of the ndgDiscovery.config file.

1. Single Sign On Service

Settings are organised under the section [NDG_SECURITY.ssoService]. Only the most important settings are described here. The rest should be left as their default values.

  • Server Address settings - the single sign on service relies on the ability to serve the Browse interface over http and https. This can be configured with Apache - see later. The sslServer setting should be the same as the server setting in the with the exception that the protocol is https e.g.

sslServer: https://ndgbeta.badc.rl.ac.uk
server: http://ndgbeta.badc.rl.ac.uk
  • Security web service addresses. - Put in here the addresses for the local Session Manager and Attribute Authority services e.g.
sessionMgrURI: https://ndgbeta.badc.rl.ac.uk/SessionManager
attAuthorityURI: http://ndgbeta.badc.rl.ac.uk/AttributeAuthority
  • SSL Connections - the Browse interface includes a login facility as part of the Single Sign On service. If another site requests the user to login, the Single Sign On Service can check that site's SSL certificate. This is an anti-phishing measure. Include in this parameter a space separated list of all the CA certificates required to enable this site to verify trusted site login requests. For help getting these files contact Phil.
sslCACertFilePathList: certs/badc-ca.crt certs/bodc-ca.crt certs/nocs-ca.crt certs/neodaas-ca.crt

2. Gatekeeper Settings

These are set in the section [NDG_SECURITY.gatekeeper]. Settings starting with pdp should be left as their defaults.

The address of the local Attribute Authority is needed:

aaURI: http://ndgbeta.badc.rl.ac.uk/AttributeAuthority

The Gatekeeper needs information in order to checks Attribute Certificates received from the Attribute Authority: a copy of the CA certificate used by the Attribute Authority and the Distinguished Name of the Attribute Authority's X.509 certificate:

acCACertFilePathList: certs/badc-ca.crt
acIssuer: /CN=AttributeAuthority/O=NDG Security Test/OU=Site A

Set-up includes the following steps:

  • certificates are created to secure communication with security web services (WS-Security and SSL Settings)
  • the Discovery/Browse? service is set up to run over http and https Virtual Hosts and
  • parameters are configured to enable access control decisions for secure data requests.

For help contact Phil.

Secure Communication with Security Web Services

Create a certificate and private key for the Browse PDP (Policy Decision Point). For the private key:

cd /etc/ndg/ows_server/conf/certs
openssl genrsa -des3 -out browse-pdp.key 2048
chmod 400 browse-pdp.key

You will be prompted for a password to protect the file. If you don't want to password protect it, omit the -des3 argument.

Then, create a new certificate request:

openssl req -new -key browse-pdp.key -out browse-pdp.csr

You will be prompted for the fields that will make up the Distinguished Name of the certificate when it is issued. It is recommended that a Common Name is set to BrowsePDP. Organisation can be NDG and Organisation Unit, the name of your organisation. Other fields can be left blank.

E-mail the request file so that it can signed and sent back to you:

mail p.j.kershaw@rl.ac.uk -s 'Certificate Request' < browse-pdp.csr

When you receive the signed certificate copy it into /etc/ndg/ows_server/conf/certs/browse-pdp.crt. Once you have the certificate, the certificate request file browse-pdp.csr can be removed. You should also install a copy of your CA certificate and the CA certificates of all the other NDG sites that you trust. If you don't know how to get this contact Phil for help. Copy the CA certificates into /etc/ndg/ows_server/conf/certs/

Certificate files can be checked with openssl e.g. the following command will print out the Distinguished Name for the CA certificate:

openssl x509 -in cacert.crt -subject

The new discovery certificate and private key and CA certificate files should be referenced in the discovery config /etc/ndg/ows_server/conf/ndgDiscovery.config) as follows:

# WS-Security signature handler
# This is an application certificate ... (which may be a machine certificate)
# X.509 certificate sent with outbound signed messages
wssCertFilePath: /etc/ndg/ows_server/conf/certs/discovery.crt

# Private key used to sign messages
# This is an application certificate ... (which may be a machine certificate)
wssKeyFilePath: /etc/ndg/ows_server/conf/certs/discovery.key

# Password for private key - comment out if the file is not password protected
wssKeyPwd: password

# Space separated list of CA cert. files to validate certs against when
# verifying responses
wssCACertFilePathList: /etc/ndg/ows_server/conf/certs/cacert.crt

In the above, replace password with the password you set to protect the private key. If no password was set leave this field blank.

Finally, the field sslCACertFilePathList can be used to authenticate peers for SSL connections to security web services. In the current implementation this applies to the Session Manager web service. This runs over https. On a request to the Session Manager, the Discovery service can verify the Session Manager's X.509 certificate against a list of acceptable CA certificates. If the Session Manager's X.509 certificate is not issued by any of the CA certificates in the list the connection is rejected.

Virtual Hosting of the Discovery Service over http and https

Paste, the Discovery application server runs over http but pages for Single Sign On require https for the secure transfer of user credentials. One way to achieve this is to run paste on a port hidden inside the firewall exposing it to the outside using Virtual Hosting e.g. with Apache. The service running on a particular port is exposed outside on 80 (http) and 443 (https):

http://localhost:8080 -> http://your-site-discovery-url
http://localhost:8080 -> https://your-site-discovery-url

Note that the same your-site-discovery-url is used in both cases.

Example .conf file configurations for Apache2 are shown below for http and https proxies:

ServerName localhost
NameVirtualHost *:80

<VirtualHost *:80>
  DocumentRoot /var/www/html
  ServerName localhost

  # NDG Discovery
  ProxyPass / http://localhost:8080/
  ProxyPassReverse / http://localhost:8080/
  ProxyPreserveHost On
  <Proxy *>
      Order deny,allow
      Allow from all
  </Proxy>
</VirtualHost>

https Virtual Host ...

ServerName localhost
NameVirtualHost *:443

<VirtualHost *:443>
  DocumentRoot /var/www/secure
  ServerName localhost
  SSLEngine On
  SSLCertificateFile /etc/apache2/ssl/crt/server.crt
  SSLCertificateKeyFile /etc/apache2/ssl/key/server.key
  SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire

  # NDG LoginService
  ProxyPass / http://localhost:8080/
  ProxyPassReverse / http://localhost:8080/
  ProxyPreserveHost On
  <Proxy *>
      Order deny,allow
      Allow from all
  </Proxy>
</VirtualHost>

SSLCertificateFile and SSLCertificateKeyFile set the certificate and private key used for SSL connections. If you don't have these, a certificate request can be made to the NDG CA. Follow the same steps as outlined in Secure Communication with Security Web Services making sure to set the certificate Common Name to the fully qualified domain name of the server host.

Details may vary according to the version of Apache you use. Please check the relevant Apache documentation for correct settings. The example uses a redirect to localhost. To expose outside, use your-site-discovery-url.

These changes should be reflected in the discovery config file, /etc/ndg/ows_server/conf/ndgDiscovery.config. The server field should be assigned http://your-site-discovery-url and sslServer to https://your-site-discovery-url:

[DEFAULT]
#
# the following is the server on which this browse/discovery instance runs!
server: http://<your-site-discovery-url>
.
.
.
[NDG_SECURITY]
sslServer: https://<your-site-discovery-url>

Gatekeeper Settings

When a request is made for secured data an Attribute Certificate is submitted on behalf of the user. This contains a list of roles or attributes that they are entitled to. It is digitally signed by the Attribute Authority service that issued it.

The acIssuer and acCACertFilePathList fields enable the Gatekeeper to verify that your organisation's Attribute Authority signed the Attribute Certificate submitted to it.

Set the acIssuer field to the Distinguished Name of the Attribute Authority's X.509 Certificate. The Attribute Authority holds an X.509 certificate on its host machine. This is normally in /etc/ndg/security/conf/certs. To check the Distinguished Name:

openssl x509 -in aa.pem -subject

The acCACertFilePathList should be set to the CA certificate that issued the Attribute Authority's X.509 Certificate. This would be expected to be the one previously obtained in /etc/ndg/ows_server/conf/cacert.crt.

# Gatekeeper Attribute Certificate check
# Issuer - should match with the issuer element of the users Attribute
# Certificate submitted in order to gain access
acIssuer: /CN=AttributeAuthority/O=NDG/OU=YourOrganisationName

# verification of X.509 cert back to CA
acCACertFilePathList: /etc/ndg/ows_server/conf/cacert.crt

Step 11: Start the services

Starting the server is a one line command:

paster serve --reload development.ini

If that all worked, then you should be able to navigate to your server eg  http://localhost:8080 and see the message:

Welcome to your Pylons Web Application

If that worked, try navigating to the discovery service: e.g.

http://localhost:8080/discovery

If that worked try and view discovery and browse records!!

CSML services (WMS and WCS) are also available in this stack, but as they are still under development and nobody has any CSML they will be the subject of another wiki page.

Step 12: Add to boot

This script can be put in /etc/init.d and will start your service on boot. Give it a suitable name, such as 'ndgservices'.

#!/bin/sh -e
# chkconfig: 2345 99 01
# description: NERC Data Grid Discovery Service

export PYTHON_EGG_CACHE=/tmp
PID=/etc/ndg/ows_server/run/paster.pid
INI=/etc/ndg/ows_server/conf/development.ini
LOG=/etc/ndg/ows_server/logs/paster.log
PROG=/usr/local/bin/paster

status()
{
    local pid=
    
    # Get pid from PID file
    local pidFound=
    if [ -f $PID ] ; then
        pid=$(cat $PID)
        if [ -z "$pid" ]; then
            echo $"Can't get pid from pid file $PID"
            return
        fi
        pidFound=Yes
    fi

    # look for pid in listing
    for i in `pidof -o $$ -o $PPID -o %PPID -x "${PROG}"`; do
        [[ $i = $pid ]] && pidFound=Yes && break;
    done
    
    if [ -n "$pidFound" ]; then
        echo $"$prog (pid $pid) is running..."

    elif [ -f ${PID} ]; then
        echo $"$prog is dead but pid file $PID exists"
    else
        echo $"$prog is dead"
    fi
}

case "$1" in
  start)
    ${PROG} serve --daemon --reload --pid-file=$PID --log-file=$LOG $INI
    ;;
  stop)
    ${PROG} serve --stop-daemon --pid-file=$PID
    ;;
  restart)
    ${PROG} serve --stop-daemon --pid-file=$PID
    ${PROG} serve --daemon --reload --pid-file=$PID --log-file=$LOG $INI
    ;;
  status)
    status
    ;;
  *)
    echo $"Usage: $(basename $0) {start|stop|restart|status}"
    exit 1
esac

you can then start/restart/stop/check status using:

sudo sh /etc/init.d/ndgservices start
sudo sh /etc/init.d/ndgservices restart
sudo sh /etc/init.d/ndgservices stop
sudo sh /etc/init.d/ndgservices status

(in this case 'ndgservices' is the name of the script)

If your Linux distribution has the chkconfig command you can register the service:

chkconfig --add ndgservices

and then administer with

sudo service ndgservices start
sudo service ndgservices restart
sudo service ndgservices stop
sudo service ndgservices status

ISSUES LOG

Please note any problems encountered or issues here - along with the solution if you know it.

Issue 1. Numeric Install

The Numeric install doesn't always go smoothly, see step 6 for possible solution.

Issue 2. Routes not working properly.

If you try a URL such as this:

http://localhost:8080/view/badc.nerc.ac.uk__NDG-B1__dataent_COAPEC

And get a "Not Found, The resource could not be found." error then that indicates a problem with the routes module.

This problem occurs with routes version 1.6.3. Upgrading to 1.7 should solve the problem. Use the -U flag in easy_install to upgrade:

easy_install -U routes

Issue 3. Zombies

When trying to set up step 12 (run at boot) I was getting lots of problems with zombie processes that I couldn't kill off. It turned out that the paster.pid file was not getting written out. If this happens edit the init script and change the location of the paster.pid file to somewhere writeable (or change the permissions of it's current location).