Changes between Version 9 and Version 10 of CowsFramework/CowsInstallation/MigrationToPython2.6


Ignore:
Timestamp:
14/11/10 09:14:26 (9 years ago)
Author:
spascoe
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • CowsFramework/CowsInstallation/MigrationToPython2.6

    v9 v10  
    1 = COWS Migration to Python 2.6 and Pylons 1.0 = 
    2  
    3 [[PageOutline]] 
    4  
    5 == Introduction == 
    6  
    7 In order to keep the COWS stack (cows + cowsserver + cowsclient) synchronised with our security solution we needed to update the stack to work with Python 2.6. The primary obstacle was updated the Pylons-related code due to a number of minor (but significant) changes introduced between Pylons 0.9.6 and 0.9.7. These changes are explained in the following Pylons documentation page: 
    8  
    9  http://wiki.pylonshq.com/pages/viewpage.action?pageId=11174779 
    10  
    11 This page documents: 
    12  
    13  * OUTSTANDING ISSUES WE NEED TO LOOK INTO REGARDING THE MIGRATION to Pylons 1.0 
    14  * changes made in COWS, COWS Server and COWS Client to update to Pylons 1.0. 
    15  
    16 During this process various dependencies were also updated and documented. A separate page documents the [wiki:CowsFramework/CowsInstallation/CowsPython2.6 Dependencies and Installation of the COWS stack]. 
    17  
    18 == OUTSTANDING ISSUES WE NEED TO LOOK INTO REGARDING THE MIGRATION to Pylons 1.0 == 
    19  
    20 The cows, cowsserver and cowsclient code has been migrated to python 2.6. However, there are a number of controllers and items that '''still need further testing''' that may, or may not, have been broken in the pre-migrated version. There are also a few changes that have been made that require validation to ensure the behaviour is appropriate. This section outlines these issues. Once resolved they should be either documented or removed from the list below. 
    21  
    22 === Untested controllers in pylons apps === 
    23  
    24 '''demo controller in cowsserver broken''' 
    25  
    26 The {{{/<dataset>/demo}}} routes point to the '''demo''' controller. At present this seems to fail because ajax is not set up right. 
    27  
    28 '''Other controllers in cowsserver and cowsclient''' 
    29  
    30 There are a number of controllers in both cowsserver and cowsclient. These need to be reviewed so that we: 
    31  
    32  * delete any test/unused controllers 
    33  * test any required controllers and explain their usage context 
    34  
    35 === cowsclient get_figure broken === 
    36  
    37 On the cows client there the {{{wmsviz.get_figure()}}} action is broken for the following file formats: 
    38  
    39  * SVG 
    40  * EPS 
    41  
    42 These are called by the web app generated by the wmsviz controller when you click the "Make Figure" button in the right-hand panel. 
    43  
    44 === Basemap problem with Transverse Mercator projection === 
    45  
    46 One of the cowsserver dependencies is [browse:qesdi/geoplot]. Within geoplot, an instance is created of the '''mpl_toolkit.basemap.Basemap''' class with the "tmerc" (Transverse Mercator) projection. The actual code is (see: http://proj.badc.rl.ac.uk/ndg/browser/qesdi/geoplot/trunk/lib/geoplot/grid_builder_national.py#L32): 
    47  
    48 {{{ 
    49 32      nationalGrid = basemap.Basemap(projection='tmerc', 
    50 33                                     lon_0=trueOrigin[0], 
    51 34                                     lat_0=trueOrigin[1], 
    52 35                                     llcrnrlon=trueOrigin[0], llcrnrlat=trueOrigin[1], 
    53 36                                     urcrnrlon=10, urcrnrlat=10, 
    54 37                                     resolution=None) 
    55 }}} 
    56  
    57 This failed because the implicit {{{rsphere}}} argument is set to the radius of the earth but as of Basemap 1.0.0 needs to provide a tuple/list of two values representing the major and minor axis of the sphere. 
    58  
    59 >> See solution below for more info and what we decided to do. 
    60  
    61 '''SOLUTION TO 'tmerc' PROJECTION PROBLEM''' 
    62  
    63 The [http://matplotlib.sourceforge.net/basemap/doc/html/api/basemap_api.html Basemap documentation] explains the use of the Basemap class: 
    64  
    65 {{{ 
    66 class: mpl_toolkits.basemap.Basemap(llcrnrlon=None, llcrnrlat=None, urcrnrlon=None, urcrnrlat=None, llcrnrx=None, llcrnry=None, urcrnrx=None, urcrnry=None, width=None, height=None, projection='cyl', resolution='c', area_thresh=None, rsphere=6370997.0, lat_ts=None, lat_1=None, lat_2=None, lat_0=None, lon_0=None, lon_1=None, lon_2=None, no_rot=False, suppress_ticks=True, satellite_height=35786000, boundinglat=None, fix_aspect=True, anchor='C', ax=None) 
    67 }}} 
    68  
    69 Within our geoplot library we create an instance as follows: 
    70  
    71 {{{ 
    72 # National Grid parameters 
    73 trueOrigin = (358.0, 49.0) 
    74  
    75 # Had to add new parameter for matplotlib 1.0.0 
    76 rsphere = [6370997, 1] 
    77  
    78 # Create a basemap with the origin at the true origin. 
    79 nationalGrid = basemap.Basemap(projection = 'tmerc', 
    80                                lon_0 = trueOrigin[0], 
    81                                lat_0 = trueOrigin[1], 
    82                                llcrnrlon = trueOrigin[0], llcrnrlat = trueOrigin[1], 
    83                                urcrnrlon = 10, urcrnrlat = 10, 
    84                                resolution = None, 
    85                                rsphere = rsphere) 
    86 }}} 
    87  
    88 However, we need to understand why we are defining the '''rsphere''' argument as {{{[6370997, 1]}}}. 
    89  
    90 The Basemap documentation says: 
    91  
    92 '''rsphere:''' radius of the sphere used to define map projection (default 6370997 meters, close to the arithmetic mean radius of the earth). If given as a sequence, the first two elements are interpreted as the radii of the major and minor axes of an ellipsoid. Note: sometimes an ellipsoid is specified by the major axis and an inverse flattening parameter (if). The minor axis (b) can be computed from the major axis (a) and the inverse flattening parameter using the formula if = a/(a-b). 
    93  
    94 So, its settings can be (as integers or floats): 
    95  
    96  1. Radius of the sphere 
    97  2. (radius of major axis, radius of minor axis) - assuming an ellipsoid 
    98  3. (radius of major axis, inverse flattening parameter) - assuming an ellipsoid 
    99  
    100 So, which to use? If we try using the default {{{6370997}}} then we get a segmentation fault. So, if we look up transverse mercator on the web we find info such as: 
    101  
    102  http://www.answers.com/topic/transverse-mercator-projection 
    103  
    104 Which discusses how the major and minor axes are define, and links to another page on the "whole earth ellipsoid": 
    105  
    106  http://www.answers.com/topic/earth-ellipsoid 
    107  
    108 On this page there is a table of possible values for Equatorial (major) axis, Polar (minor) axis and Inverse flatenning (1/f) parameter. The most current is the WGS84 ellipsoid definition of: 
    109  
    110  * Name:WGS 1984 
    111  * Equatorial axis (m): 6,378,137.00 
    112  * Polar axis (m): 6,356,752.3142       
    113  * Inverse flattening (1/f): 298.257223563 
    114  * WGS 1984:    
    115  
    116 From another page (http://www.answers.com/topic/reference-ellipsoid): 
    117  
    118  {{{Currently the most common reference ellipsoid used, and that used in the context of the Global Positioning System, is WGS 84.}}} 
    119  
    120 So, if we plug in the WGS 84 reference ellipsoid to Basemap using the following script: 
    121  
    122 {{{ 
    123 #!/usr/bin/env python 
    124  
    125 # Get inputs 
    126 import sys 
    127 major_axis = float(sys.argv[1]) 
    128 arg2 = float(sys.argv[2]) 
    129  
    130 from mpl_toolkits.basemap import Basemap 
    131  
    132 # National Grid parameters 
    133 trueOrigin = (358.0, 49.0) 
    134  
    135 # Added new parameter for matplotlib 1.0.0 
    136 rsphere = [major_axis, arg2] 
    137  
    138 # Create a basemap with the origin at the true origin. 
    139 nationalGrid = Basemap(projection = 'tmerc', 
    140                                lon_0 = trueOrigin[0], 
    141                                lat_0 = trueOrigin[1], 
    142                                llcrnrlon = trueOrigin[0], llcrnrlat = trueOrigin[1], 
    143                                urcrnrlon = 10, urcrnrlat = 10, 
    144                                resolution = None, 
    145                                rsphere = rsphere) 
    146 }}} 
    147  
    148 So, run the test: 
    149  
    150 {{{ 
    151 $ python test.py 6378137.00 6356752.3142 
    152 }}} 
    153  
    154 This works fine. It seems reasonable for us to use the WGS 84 reference ellipsoid. 
    155  
    156 So we'll use: 
    157  
    158 {{{ 
    159 rsphere = [6378137.00, 6356752.3142] 
    160 }}} 
    161  
    162 The solution was encoded into the trunk of geoplot at: 
    163  
    164  
    165 This appears to be backwards compatible as well with previous versions (0.9.8) of matplotlib/basemap. We have not seen any strangeness in plotting based on this code. 
    166  
    167 === Possible alternative to metacarta coastlines === 
    168  
    169 Metacarta web services (used by COWS to serve the coastline maps) have experienced problems recently. An alternative coastline dataset could be: 
    170  
    171  * http://coastwatch.pfeg.noaa.gov/erddap/wms/usgsCeSS11.1/request?LAYERS=Coastlines&TRANSPARENT=true&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&FORMAT=image%2Fpng&SRS=EPSG%3A4326&BBOX=-180,-90,0,90&WIDTH=256&HEIGHT=256 
    172  
    173 ----- 
    174  
    175 == COWS modifications == 
    176  
    177 The changes made to the cows library are documented in the [http://proj.badc.rl.ac.uk/ndg/changeset?new=7492%40cows%2Fbranches%2Fmigrate-py26-pylons10&old=7342%40cows%2Fbranches%2Fmigrate-py26-pylons10 diff between the two versions]. 
    178  
    179 The changes all related to the following change: 
    180  
    181 {{{ 
    182 Replaced use of "routes.url_for" function with the global "pylons.url" 
    183 function for compatibility with pylons 1.0. 
    184 }}} 
    185  
    186 == COWS Server modifications == 
    187  
    188 The COWS server changes are documented in the [http://proj.badc.rl.ac.uk/ndg/changeset?new=7490%40cowsserver%2Fbranches%2Fmigrate-py26-pylons10&old=7340%40cowsserver%2Fbranches%2Fmigrate-py26-pylons10 diff between the two versions]. 
    189  
    190 These differences covered the following updates relating to Pylons changes: 
    191  
    192 '''1. config/middleware.py file changes'''  
    193  
    194 {{{ 
    195 config/middleware.py file updated to mirror standard pylons 1.0 version. 
    196 Also removed the switch for compatibility with pylons versions before 
    197 version 0.9.7. 
    198 }}} 
    199  
    200 '''2. Setting of pylons global "c" variable config attributes (to retain previous behaviour)''' 
    201  
    202 {{{ 
    203 Included additional config args introduced in pylons 0.9.7 to allow the 
    204 global "c" variable to maintain access to global properties and to not 
    205 throw AttributeError exceptions when an attribute is not found. See: 
    206  
    207 http://wiki.pylonshq.com/pages/viewpage.action?pageId=11174779 
    208 }}} 
    209  
    210 '''3. Modification of config/routing.py syntax and "map" attributes''' 
    211  
    212 {{{ 
    213 Updated routes to pylons 1.0 requirement using new Routes syntax. 
    214 Also added the lines: 
    215  
    216     # We need minimization switched on 
    217     map.minimization = True 
    218     map.explicit = False 
    219  
    220 To handle enable "implicit" routing, see: 
    221 http://routes.groovie.org/manual.html#implicit-defaults-and-route-memory  
    222 }}} 
    223  
    224 '''4. Replacing "routes.url_for" with "pylons.url" calls''' 
    225  
    226 {{{ 
    227 Replaced use of "routes.url_for" function with the global "pylons.url" 
    228 function for compatibility with pylons 1.0. 
    229 }}} 
    230  
    231 == COWS Client modifications == 
    232  
    233 The COWS client changes are documented in the [http://proj.badc.rl.ac.uk/ndg/changeset?new=7502%40cowsclient%2Fbranches%2Fmigrate-py26-pylons10&old=7344%40cowsclient%2Fbranches%2Fmigrate-py26-pylons10 diff between the two versions]. 
    234  
    235 These differences covered the following updates relating to Pylons changes: 
    236  
    237 '''1. config/middleware.py file changes'''  
    238  
    239 {{{ 
    240 config/middleware.py file updated to mirror standard pylons 1.0 version. 
    241 Also removed the switch for compatibility with pylons versions before 
    242 version 0.9.7. 
    243 }}} 
    244  
    245 '''2. Setting of pylons global "c" variable config attributes (to retain previous behaviour)''' 
    246  
    247 {{{ 
    248 Included additional config args introduced in pylons 0.9.7 to allow the 
    249 global "c" variable to maintain access to global properties and to not 
    250 throw AttributeError exceptions when an attribute is not found. See: 
    251  
    252 http://wiki.pylonshq.com/pages/viewpage.action?pageId=11174779 
    253 }}} 
    254  
    255 '''3. Modification of config/routing.py syntax and "map" attributes''' 
    256  
    257 {{{ 
    258 Updated routes to pylons 1.0 requirement using new Routes syntax. 
    259 Also added the lines: 
    260  
    261     # We need minimization switched on 
    262     map.minimization = True 
    263     map.explicit = False 
    264  
    265 To handle enable "implicit" routing, see: 
    266 http://routes.groovie.org/manual.html#implicit-defaults-and-route-memory  
    267 }}} 
    268  
    269 '''4. Replacing "routes.url_for" with "pylons.url" calls''' 
    270  
    271 {{{ 
    272 Replaced use of "routes.url_for" function with the global "pylons.url" 
    273 function for compatibility with pylons 1.0. 
    274 }}} 
    275  
    276 '''5. Escaping "<" character in Javascript within HTML template''' 
    277  
    278 {{{ 
    279 Added CDATA section to escape the "<" character in javascript within an 
    280 HTML file. This stops Genshi escaping the "<" character into "&lt;" and 
    281 interpreting it as HTML instead of Javascript. 
    282 }}} 
     1See http:cows.badc.rl.ac.uk/CowsFramework/MigrationToPython2.6