wiki:CowsFramework/MigrationToPython2.6

COWS Migration to Python 2.6 and Pylons 1.0

Introduction

In order to keep the COWS stack (cows + cowsserver + cowsclient) synchronised with our security solution we needed to update the stack to work with Python 2.6. The primary obstacle was updated the Pylons-related code due to a number of minor (but significant) changes introduced between Pylons 0.9.6 and 0.9.7. These changes are explained in the following Pylons documentation page:

 http://wiki.pylonshq.com/pages/viewpage.action?pageId=11174779

This page documents:

  • OUTSTANDING ISSUES WE NEED TO LOOK INTO REGARDING THE MIGRATION to Pylons 1.0
  • changes made in COWS, COWS Server and COWS Client to update to Pylons 1.0.

During this process various dependencies were also updated and documented. A separate page documents the Dependencies and Installation of the COWS stack.

OUTSTANDING ISSUES WE NEED TO LOOK INTO REGARDING THE MIGRATION to Pylons 1.0

The cows, cowsserver and cowsclient code has been migrated to python 2.6. However, there are a number of controllers and items that still need further testing that may, or may not, have been broken in the pre-migrated version. There are also a few changes that have been made that require validation to ensure the behaviour is appropriate. This section outlines these issues. Once resolved they should be either documented or removed from the list below.

Untested controllers in pylons apps

demo controller in cowsserver broken

The /<dataset>/demo routes point to the demo controller. At present this seems to fail because ajax is not set up right.

Other controllers in cowsserver and cowsclient

There are a number of controllers in both cowsserver and cowsclient. These need to be reviewed so that we:

  • delete any test/unused controllers
  • test any required controllers and explain their usage context

cowsclient get_figure broken

On the cows client there the wmsviz.get_figure() action is broken for the following file formats:

  • SVG
  • EPS

These are called by the web app generated by the wmsviz controller when you click the "Make Figure" button in the right-hand panel.

Basemap problem with Transverse Mercator projection

One of the cowsserver dependencies is [browse:qesdi/geoplot]. Within geoplot, an instance is created of the mpl_toolkit.basemap.Basemap class with the "tmerc" (Transverse Mercator) projection. The actual code is (see:  http://proj.badc.rl.ac.uk/ndg/browser/qesdi/geoplot/trunk/lib/geoplot/grid_builder_national.py#L32):

32	nationalGrid = basemap.Basemap(projection='tmerc',
33	                               lon_0=trueOrigin[0],
34	                               lat_0=trueOrigin[1],
35	                               llcrnrlon=trueOrigin[0], llcrnrlat=trueOrigin[1],
36	                               urcrnrlon=10, urcrnrlat=10,
37	                               resolution=None)

This failed because the implicit rsphere argument is set to the radius of the earth but as of Basemap 1.0.0 needs to provide a tuple/list of two values representing the major and minor axis of the sphere.

See solution below for more info and what we decided to do.

SOLUTION TO 'tmerc' PROJECTION PROBLEM

The  Basemap documentation explains the use of the Basemap class:

class: mpl_toolkits.basemap.Basemap(llcrnrlon=None, llcrnrlat=None, urcrnrlon=None, urcrnrlat=None, llcrnrx=None, llcrnry=None, urcrnrx=None, urcrnry=None, width=None, height=None, projection='cyl', resolution='c', area_thresh=None, rsphere=6370997.0, lat_ts=None, lat_1=None, lat_2=None, lat_0=None, lon_0=None, lon_1=None, lon_2=None, no_rot=False, suppress_ticks=True, satellite_height=35786000, boundinglat=None, fix_aspect=True, anchor='C', ax=None)

Within our geoplot library we create an instance as follows:

# National Grid parameters
trueOrigin = (358.0, 49.0)

# Had to add new parameter for matplotlib 1.0.0
rsphere = [6370997, 1]

# Create a basemap with the origin at the true origin.
nationalGrid = basemap.Basemap(projection = 'tmerc',
                               lon_0 = trueOrigin[0],
                               lat_0 = trueOrigin[1],
                               llcrnrlon = trueOrigin[0], llcrnrlat = trueOrigin[1],
                               urcrnrlon = 10, urcrnrlat = 10,
                               resolution = None,
                               rsphere = rsphere)

However, we need to understand why we are defining the rsphere argument as [6370997, 1].

The Basemap documentation says:

rsphere: radius of the sphere used to define map projection (default 6370997 meters, close to the arithmetic mean radius of the earth). If given as a sequence, the first two elements are interpreted as the radii of the major and minor axes of an ellipsoid. Note: sometimes an ellipsoid is specified by the major axis and an inverse flattening parameter (if). The minor axis (b) can be computed from the major axis (a) and the inverse flattening parameter using the formula if = a/(a-b).

So, its settings can be (as integers or floats):

  1. Radius of the sphere
  2. (radius of major axis, radius of minor axis) - assuming an ellipsoid
  3. (radius of major axis, inverse flattening parameter) - assuming an ellipsoid

So, which to use? If we try using the default 6370997 then we get a segmentation fault. So, if we look up transverse mercator on the web we find info such as:

 http://www.answers.com/topic/transverse-mercator-projection

Which discusses how the major and minor axes are define, and links to another page on the "whole earth ellipsoid":

 http://www.answers.com/topic/earth-ellipsoid

On this page there is a table of possible values for Equatorial (major) axis, Polar (minor) axis and Inverse flatenning (1/f) parameter. The most current is the WGS84 ellipsoid definition of:

  • Name:WGS 1984
  • Equatorial axis (m): 6,378,137.00
  • Polar axis (m): 6,356,752.3142
  • Inverse flattening (1/f): 298.257223563
  • WGS 1984:

From another page ( http://www.answers.com/topic/reference-ellipsoid):

Currently the most common reference ellipsoid used, and that used in the context of the Global Positioning System, is WGS 84.

So, if we plug in the WGS 84 reference ellipsoid to Basemap using the following script:

#!/usr/bin/env python

# Get inputs
import sys
major_axis = float(sys.argv[1])
arg2 = float(sys.argv[2])

from mpl_toolkits.basemap import Basemap

# National Grid parameters
trueOrigin = (358.0, 49.0)

# Added new parameter for matplotlib 1.0.0
rsphere = [major_axis, arg2]

# Create a basemap with the origin at the true origin.
nationalGrid = Basemap(projection = 'tmerc',
                               lon_0 = trueOrigin[0],
                               lat_0 = trueOrigin[1],
                               llcrnrlon = trueOrigin[0], llcrnrlat = trueOrigin[1],
                               urcrnrlon = 10, urcrnrlat = 10,
                               resolution = None,
                               rsphere = rsphere)

So, run the test:

$ python test.py 6378137.00 6356752.3142

This works fine. It seems reasonable for us to use the WGS 84 reference ellipsoid.

So we'll use:

rsphere = [6378137.00, 6356752.3142]

The solution was encoded into the trunk of geoplot at:

This appears to be backwards compatible as well with previous versions (0.9.8) of matplotlib/basemap. We have not seen any strangeness in plotting based on this code.

Possible alternative to metacarta coastlines

Metacarta web services (used by COWS to serve the coastline maps) have experienced problems recently. An alternative coastline dataset could be:


COWS modifications

The changes made to the cows library are documented in the  diff between the two versions.

The changes all related to the following change:

Replaced use of "routes.url_for" function with the global "pylons.url"
function for compatibility with pylons 1.0.

COWS Server modifications

The COWS server changes are documented in the  diff between the two versions.

These differences covered the following updates relating to Pylons changes:

1. config/middleware.py file changes

config/middleware.py file updated to mirror standard pylons 1.0 version.
Also removed the switch for compatibility with pylons versions before
version 0.9.7.

2. Setting of pylons global "c" variable config attributes (to retain previous behaviour)

Included additional config args introduced in pylons 0.9.7 to allow the
global "c" variable to maintain access to global properties and to not
throw AttributeError exceptions when an attribute is not found. See:

http://wiki.pylonshq.com/pages/viewpage.action?pageId=11174779

3. Modification of config/routing.py syntax and "map" attributes

Updated routes to pylons 1.0 requirement using new Routes syntax.
Also added the lines:

    # We need minimization switched on
    map.minimization = True
    map.explicit = False

To handle enable "implicit" routing, see:
http://routes.groovie.org/manual.html#implicit-defaults-and-route-memory 

4. Replacing "routes.url_for" with "pylons.url" calls

Replaced use of "routes.url_for" function with the global "pylons.url"
function for compatibility with pylons 1.0.

COWS Client modifications

The COWS client changes are documented in the  diff between the two versions.

These differences covered the following updates relating to Pylons changes:

1. config/middleware.py file changes

config/middleware.py file updated to mirror standard pylons 1.0 version.
Also removed the switch for compatibility with pylons versions before
version 0.9.7.

2. Setting of pylons global "c" variable config attributes (to retain previous behaviour)

Included additional config args introduced in pylons 0.9.7 to allow the
global "c" variable to maintain access to global properties and to not
throw AttributeError exceptions when an attribute is not found. See:

http://wiki.pylonshq.com/pages/viewpage.action?pageId=11174779

3. Modification of config/routing.py syntax and "map" attributes

Updated routes to pylons 1.0 requirement using new Routes syntax.
Also added the lines:

    # We need minimization switched on
    map.minimization = True
    map.explicit = False

To handle enable "implicit" routing, see:
http://routes.groovie.org/manual.html#implicit-defaults-and-route-memory 

4. Replacing "routes.url_for" with "pylons.url" calls

Replaced use of "routes.url_for" function with the global "pylons.url"
function for compatibility with pylons 1.0.

5. Escaping "<" character in Javascript within HTML template

Added CDATA section to escape the "<" character in javascript within an
HTML file. This stops Genshi escaping the "<" character into "&lt;" and
interpreting it as HTML instead of Javascript.