wiki:Software/MSI/ConfigurationNeptune

Version 37 (modified by pjkersha, 10 years ago) (diff)

--

Neptune Configuration

Neptune is the test server for NDG3 (MSI).

$ cat /etc/issue

Welcome to openSUSE 10.3 (X86-64) - Kernel \r (\l).

Neptune has an alias ndg3beta.badc.rl.ac.uk.

Known Issues

  • ssh login fails with:
    $ ssh neptune.badc.rl.ac.uk
    Password:
    Permissions on the password database may be too restrictive.
    
    Password:
    
    ... indicates the store mount has failed. Contact SDDCS
  • wildly inaccurate dates causing unpredictable behaviour. Contact SDDCS

Python Configuration

System default is Python 2.5 in /usr/bin. Under SuSE, the site package location is customised to /usr/local/lib64/python2.5/site-packages with /usr/lib64/python2.5/distutils/distutils.cfg.

Application packages will be installed separately to avoid version conflicts and maintenance problems with a single package area. virtualenv or zc.buildout could achieve this. virtualenv is easy to set-up with mod_wsgi - see Apache Configuration. zc.buildout enables overriding control over package versions to define a package and version combination to make a stable deployment. zc.buildout  collective.recipe.modwsgi enables integration with mod_wsgi.

Virtualenv

virtualenv bootstrap:

$ virtualenv --no-site-packages myenv

... failed with this message:

TEST FAILED: /usr/local/lib64/python2.5/site-packages/ does NOT support .pth files 

This is a known problem with SuSE:

 http://groups.google.com/group/python-virtualenv/browse_thread/thread/aa69f8b738d23652

This discussion suggests commenting out the prefix setting in /usr/lib64/python2.5/distutils/distutils.cfg, but a less intrusive option is to override the setting by creating an  alternative config file setup.cfg or ~/.pydistutils.cfg:

$ cat > ./setup.cfg
[build_py]
optimize=0

[install]
prefix=/mypath/myenv
optimize=0

Then ...

$ virtualenv --no-site-packages myenv

Move the setup.cfg file to within the virtualenv directory so that it doesn't interfere with other components: $ mv ./setup.cfg ./myenv Install setuptools cding to myenv directory first to ensure that setup.cfg is picked up: {{{ $ cd ./myenv $ wget  http://peak.telecommunity.com/dist/ez_setup.py $ ./bin/python ./ez_setup.py }}}

zc.buildout

See the Discovery service browser interface section for details.

Apache Configuration

See: Apache Configuration page.

Application Configuration

Discovery Browser Interface

The discovery service has been deployed with a buildout script (as of 04/06/2009). The configuration is in /usr/local/ndg-discovery. The procedure to set up was:

  1. Install zc.buildout and  collective.recipe.modwsgi:
    $ sudo /usr/local/bin/easy_install zc.buildout
    $ sudo /usr/local/bin/easy_install collective.recipe.modwsgi
    
  2. The Apache config needs altering to add a mount point for the Discovery service:
    WSGIScriptAlias /services /srv/www/wsgi_scripts/discovery.wsgi
    
  3. Create the buildout.cfg /usr/local/ndg-discovery:
    [buildout]
    parts = DiscoveryServiceProGlueMirror
    
    # Configuration mirroring eggs as currently deployed on proglue
    [DiscoveryServiceProGlueMirror]
    recipe = collective.recipe.modwsgi
    
    eggs = ows_server==0.0.0dev_r5354
            ows_common==0.1dev_r2969
            ndgUtils==0.1.2.dev_r4896
            csml==2.0b_r3046
            cdat_lite>=4.1.2_0.2.5
    #       cdat_lite==4.1.2_0.2.5
            Pylons==0.9.6.1
            WebHelpers==0.3.2
    config-file = ${buildout:directory}/service.ini
    find-links = http://ndg.nerc.ac.uk/dist
            http://ndg.nerc.ac.uk/dist/archivedcsml
    
    The DiscoveryServiceProGlueMirror part mirrors the configuration on proglue:
    • Explicit Pylons and WebHelpers versions were set to avoid webhelpers 'auto_link' AttributeError.
    • cdat_lite should be set to version 4 but this wouldn't install. Version 5 installs but there is a known error with cdms imports. This will be fixed with an upgrade to the latest version of discovery.
  4. To generate the configuration run buildout from /usr/local/ndg-discovery.
    $ /usr/local/bin/buildout
    
    This will create a WSGI script in ./parts/DiscoveryServiceProGlueMirror/wsgi
  5. The script as generated could be improved to enable logging by adding a line to include fileConfig from paste.script.util.logging_config e.g.:
    from paste.deploy import loadapp
    from paste.script.util.logging_config import fileConfig
    
    configFilePath = '/usr/local/myapp/service.ini'
    fileConfig(configFilePath)
    
    application = loadapp('config:%s' % configFilePath)
    
    This could be done by customising the collective.recipe.modwsgi recipe. As an interim measure. a Makefile makes the changes using an ugly set of sed commands :) The Makefile also includes targets to install the script into the right area for Apache to pick up:
    WSGI_DIR=/srv/www/wsgi_scripts
    WSGI_SCRIPT_NAME=discovery.wsgi
    WSGI_SCRIPT_IN_FILE=./parts/DiscoveryService/wsgi
    TMP_FILE=${WSGI_SCRIPT_IN_FILE}.tmp
    
    install_wsgi: add_logging
            @echo installing WSGI script ...
            cp ${TMP_FILE} ${WSGI_DIR}/${WSGI_SCRIPT_NAME}
            @echo Done.
    
    add_logging:
            @echo Altering WSGI script file to include logging functionality ...
            @cat ${WSGI_SCRIPT_IN_FILE} | sed s/"application = loadapp(\"config:"/"from paste.script.util.logging_config import fileConfig\n\nconfigFilePath
     = \""/g |sed s/"\")"/"\"\nfileConfig(configFilePath)\n\napplication = loadapp(\"config:\"+configFilePath)"/g > ${TMP_FILE}
            @echo Done.
    
    http_proxy=http://wwwcache.rl.ac.uk:8080
    
    buildout:
            export http_proxy=${http_proxy}; /usr/local/bin/buildout
    
    To build and install then,
    $ make buildout
    $ make
    
    mod_wsgi set up in daemon mode will automatically reload the script content without the need to restart Apache.
  6. Nb. Ensure debug = false in the ini file config! Check ndgDiscovery.config server setting consistent with the mount point set by WSGIScriptAlias in the Apache config file.

Discovery Web Service

To run the Discovery backend server the following has been installed:

Apache Tomcat 6.0.18 (/usr/local/apache/apache-tomcat-6.0.18)

Tomcat management info can be found in conf/tomcat-users.xml

Apache Axis Axis2-1.4 (/usr/local/axis2/axis2-1.4) (Axis2.war deployed within tomcat)

Apache Ant 1.7.1 (/usr/local/ant)

Updates to /etc/init.d/tomcat have been made to allow tomcat to stop/start Discovery service backend. Following environment variables need to be set:

export JAVA_HOME=/usr/java/jdk1.5.0_08 export JRE_HOME=/usr/java/jdk1.5.0_08/jre export CATALINA_HOME=/usr/local/apache/apache-tomcat-6.0.18 export AXIS2_HOME=/usr/local/axis2/axis2-1.4

Updates to catalina.sh in $CATALINA_HOME/bin/ need to be made:

JAVA_OPTS="-Xmx1024m -Dhttp.proxyPort=8080 -Dhttp.proxyHost=wwwcache.rl.ac.uk -Dhttp.nonProxyHosts=\"*.rl.ac.uk|localhost\"" JRE_HOME="/usr/java/jdk1.5.0_08/jre"

Postgres database

Postgres 8.3.1 has been installed in the standard location: /usr/local/pgsql. The database data files and configuration files are stored in the 'data' subdirectory. The postgis extension has also been installed. This database is not currently being backed up.

Postgres should automatically start on reboot via /etc/init.d/postgresql file.

To start/stop/restart/reload postgres:

As linux user 'postgres':

/usr/local/pgsql/bin/pg_ctl start/stop/restart/reload/status -D /usr/local/pgsql/data

OR, As 'root':

/etc/init.d/postgresql start/stop/restart/status

Passwords can be found in the Secrets box.

COWS

Dom has installed COWS in a dedicated Python installation under /usr/local/cowsenv. This may be revised at a later date to use virtualenv or zc.buildout based install. See Apache Configuration page for details of COWS WSGI script.

Configuration settings are in /usr/local/cowsenv/services/OGCTestbed

Added:

  • gcc-fortran compiler (yast)
  • numpy-1.2.1
  • matplotlib-0.98.5.2.tar.gz
  • csml 2.5.1
  • cows 0.2.3
  • Paste 1.7.1
  • Genshi 0.5.1
  • PIL 1.1.6

Pylons/CSML/MATPLOTLIB etc:

  • numpy 1.2.1
  • matplotlib 0.98.5

Specific config for ...

  • WMS / WCS
    • WMS server
    • WCS server
    • WMS client with WCS download button

pyDAP

pyDAP 3.0 is installed in /usr/local/dap. This is being updated to use a zc.buildout configuration:

[buildout]
parts = pyDAP

# Configuration mirroring eggs as currently deployed on proglue
[pyDAP]
recipe = collective.recipe.modwsgi

eggs = ndg_security
       Pydap==3.0.b.4
       pydap.handlers.netcdf==0.4.4
       pydap.responses.netcdf==0.1
config-file = ${buildout:directory}/etc/service.ini
find-links = http://ndg.nerc.ac.uk/dist

The eggs list includes NetCDF response and handler plugins and NDG Security filter to intercept requests.

Security

There are two components:

  1. Security Services: an application running security services such as OpenID, Attribute Authority and Session Management
  2. Application Filters: handler filters which are configured with existing applications to protect them

The first is installed in it's own mod_wsgi application running over HTTPS. For the second, there are filters configured to secure COWS and pyDAP services.

Security Services

This is installed using the same technique as described above for the Discovery Service Browser interface: a buildout script installs the eggs required in /usr/local/ndg-security/eggs and creates a mod_wsgi script. A Makefile installs the script in the script location set-up for Apache mod_wsgi scripts. The script is mounted via a WSGIScriptAlias directive in the Apache config file.

[buildout]
parts = NDGSecurity

[NDGSecurity]
recipe = collective.recipe.modwsgi
interpreter = py

# SQLAlchemy is used by OpenID / Session Manager for authentication Database
eggs =
    ndg_security
    ndg_security_test
    SQLAlchemy
config-file = ${buildout:directory}/config/securityservices.ini
find-links = http://ndg.nerc.ac.uk/dist

See the README file in /usr/local/ndg-security for additional configuration information.

Application Filters

Each application secured with NDG Security is configured with security filters to intercept requests to them. These are applied by making settings in the Paste ini file for the application. This ini file extract shows how the main application is configured at the end of a WSGI pipeline fronted with Authentication and Authorisation filters:

[pipeline:main]
pipeline = AuthenticationFilter AuthorizationFilter myApp

AuthenticationFilter, AuthorizationFilter and myApp refer to sub-sections in the file where the specific settings are for each individual component.

The authentication filter is configured to invoke the OpenID Relying Party interface running in the Security Services application stack. This prompts the user to enter the OpenID for their home site.

The authorisation filter is configured with an XML policy file which sets which request URIs are to be secured.

pyDAP and COWS services are secured with this set-up.