Changes between Version 9 and Version 10 of CowsFramework/CowsInstallation/CowsPython2.6


Ignore:
Timestamp:
14/11/10 09:05:25 (9 years ago)
Author:
spascoe
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • CowsFramework/CowsInstallation/CowsPython2.6

    v9 v10  
    1 = Installing the COWS stack in Python 2.6 = 
    2  
    3 [[PageOutline]] 
    4  
    5 == Introduction == 
    6  
    7 These notes were initially developed as part of the process of porting the COWS stack (== cows + cowsserver + cowsclient) to Python 2.6 and Pylons 1.0 (see the [wiki:CowsFramework/CowsInstallation/MigrationToPython2.6 migration to python 2.6 page]. They also serve as a useful set of guidelines on how to build the COWS stack including a number of dependencies that require some manual intervention in the build process. 
    8  
    9 == COWS server and COWS client == 
    10  
    11 Since the COWS server dependencies are a superset of the COWS client dependencies you the installation procedure for the COWS server is presented in full and you can infer the subset of requirements for the COWS client. 
    12  
    13 == Installing COWS server dependencies == 
    14  
    15 The COWS Server compatible with Pylons 1.0 and Python 2.6 has the following dependencies. This list was produced using the 
    16 {{{pip freeze}}} tool: 
    17  
    18 {{{ 
    19   numpy==1.3.0 
    20   pycairo==1.2.2 
    21   PIL==1.1.7 
    22   Shapely==1.2.5   
    23   cdat-lite==5.2-1 
    24   geoplot>=0.4.0 
    25   csml==2.7.13   
    26   Pylons==1.0 
    27   Beaker==1.5.4 
    28   decorator==3.2.0  
    29   FormEncode==1.2.3dev 
    30   Mako==0.3.4 
    31   MarkupSafe==0.11   
    32   nose==0.11.4   
    33   Paste==1.7.5.1 
    34   PasteDeploy==1.3.4 
    35   PasteScript==1.7.3 
    36   Routes==1.12.3 
    37   simplejson==2.1.1 
    38   Tempita==0.5dev 
    39   WebError==0.10.2 
    40   WebHelpers==1.2 
    41   WebOb==1.0 
    42   WebTest==1.2.2  
    43   Genshi==0.6 
    44   OWSLib==0.3.1      
    45   matplotlib==1.0.0   
    46   basemap==1.0 
    47   mock==0.7.0b2 
    48   wsgiref==0.1.2   
    49    
    50 # For testing only 
    51 #  libxml2-python==2.6.21 
    52 #  libxml2dom==0.4.7 
    53 }}} 
    54  
    55 The rest of this page explains how to  
    56  
    57 == COWS Server (and COWS and COWS client) installation (with dependencies) == 
    58  
    59 These instructions suggest installing the COWS server into a [http://pypi.python.org/pypi/virtualenv virtualenv] self-contained python environment. They also suggest the use of [http://pypi.python.org/pypi/pip pip] as the installation tool. Alternatively, buildout can be used. 
    60  
    61 Note that not everything builds automatically and there are various stages when environment variables need to be set to pick up appropriate versions of packages. 
    62  
    63 I have tried to order the dependency list correctly to avoid having to re-build any packages (hopefully!). 
    64  
    65 === 1. Non-python packages === 
    66  
    67 '''NetCDF''' 
    68  
    69 Version: 3.6.3 
    70  
    71 Manually install NetCDF libraries (Fortran not needed): 
    72  
    73 {{{ 
    74 $ cd $HOME 
    75 $ mkdir -p external/netcdf/src 
    76 $ cd external/netcdf/src 
    77 $ wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-3.6.3.tar.gz 
    78 $ tar xvzf netcdf-3.6.3.tar.gz 
    79 $ cd netcdf-3.6.3/ 
    80 $ ./configure --enable-shared --disable-f90 --disable-f77 --prefix=$HOME/external/netcdf 
    81 $ make 
    82 $ make install 
    83 $ make check 
    84 }}} 
    85  
    86 '''Cairo''' 
    87  
    88 Version: 1.2.4 
    89  
    90 Probably exists on any server, but can be built by Yast or package manager. 
    91  
    92 '''GEOS''' 
    93  
    94 GEOS is needed for libgeos_c.so file for python, used in matplotlib, basemap and shapely. 
    95  
    96 {{{ 
    97 $ cd 
    98 $ cd external/netcdf/src 
    99 $ wget http://download.osgeo.org/geos/geos-3.2.2.tar.bz2 
    100 $ bunzip2 geos-3.2.2.tar.bz2 
    101 $ tar xvf geos-3.2.2.tar 
    102 $ cd geos-3.2.2 
    103 $ mkdir $HOME/external/geos 
    104 $ ./configure --prefix=$HOME/external/geos 
    105 $ make 
    106 $ make install 
    107 }}} 
    108  
    109 === 2. Python packages === 
    110  
    111 First activate the virtualenv environment: 
    112  
    113 {{{ 
    114 $ cd $HOME 
    115 $ . ENV/bin/activate 
    116 }}} 
    117  
    118 Most of the following packages was installed by typing: 
    119  
    120 {{{ 
    121 $ cd $HOME 
    122 $ pip install <package>==<version> 
    123  
    124 # E.g. 
    125  
    126 $ pip install numpy=1.3.0 
    127 }}} 
    128  
    129 Now start installing... 
    130  
    131 '''Numpy''' 
    132  
    133 Version: 1.3.0 
    134  
    135 pip installed egg: numpy-1.3.0-py2.6-linux-x86_64.egg 
    136  
    137 {{{ 
    138 pip install numpy==1.3.0 
    139 }}} 
    140  
    141 '''CdatLite''' 
    142  
    143 Version: 5.2_1 
    144  
    145 pip installed egg: cdat_lite-5.2_1-py2.6-linux-x86_64.egg 
    146  
    147 {{{ 
    148 $ export NETCDF_HOME=$HOME/external/netcdf 
    149 $ pip install cdat_lite==5.2_1 
    150 }}} 
    151  
    152 '''CSML''' 
    153  
    154 Version: 2.7.13 
    155  
    156 easy_install installed egg: csml-2.7.13-py2.6.egg 
    157  
    158 {{{ 
    159 easy_install --install-dir=ENV/lib/python2.6/site-packages http://ndg.nerc.ac.uk/dist/csml-2.7.13-py2.6.egg 
    160 }}} 
    161  
    162 '''Pycairo''' 
    163  
    164 Version: 1.2.2 
    165  
    166 Needed to manually install this to use compatible version with system cairo version. 
    167  
    168 Needed to build pycairo compatible with release on westerly of cairo (1.2.4) so used 1.2.2 of pycairo. Needed to download manually rather than use pip (because too old?). 
    169  
    170 {{{ 
    171 $ cd $HOME/external/src 
    172 $ wget http://cairographics.org/releases/pycairo-1.2.2.tar.gz 
    173 $ tar xvzf pycairo-1.2.2.tar.gz 
    174 $ cd pycairo-1.2.2/ 
    175 $ # Need to make this dir for it to install OK 
    176 $ mkdir $HOME/python26/lib/pkgconfig 
    177 $ python setup.py build 
    178 $ python setup.py install 
    179 }}} 
    180  
    181 Test with: 
    182  
    183 {{{ 
    184 $ python -c 'import cairo' 
    185 }}} 
    186  
    187 '''matplotlib''' 
    188  
    189 Version: 1.0.0 
    190  
    191 Using egg: matplotlib-1.0.0-py2.6-linux-x86_64.egg 
    192  
    193 Cannot install with PIP. Need to extract the egg and build locally, as instructed on wiki page at: 
    194  
    195  http://proj.badc.rl.ac.uk/dcip/wiki/UkcipDdp/Deployment/MatplotlibInstall 
    196  
    197 {{{ 
    198 $ # We'll extract an editable version of matplotlib to ./matplotlib 
    199 $ cd $HOME/external/src 
    200 $ easy_install -eb . matplotlib==1.0.0 
    201 $ # Note the "pip install --src=. matplotlib==1.0.0" should also work 
    202 $ cd matplotlib 
    203 $ # Create a setup config file 
    204 $ cp setup.cfg.template setup.cfg 
    205 }}} 
    206  
    207 Now edit the config file so it contains the following lines: 
    208  
    209 {{{ 
    210 #tag_svn_revision = 1 
    211 --- 
    212 backend = Cairo 
    213 --- 
    214 # 
    215 # Added datapath to point to mpl-data dir 
    216 datapath=/disks/westerly1/bull/ENV/etc/mpl-data 
    217 }}} 
    218  
    219 Make the {{{~/ENV/etc}}} dir to put the matplotlib data files in: 
    220  
    221 {{{ 
    222 $ mkdir ~/ENV/etc 
    223 }}} 
    224  
    225 Now create the egg: 
    226  
    227 {{{ 
    228 $ python setupegg.py bdist_egg 
    229 }}} 
    230  
    231 Now copy the new egg to site-packages: 
    232  
    233 {{{ 
    234 $ cp dist/*.egg $HOME/ENV/lib/python2.6/site-packages/ 
    235 $ export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/matplotlib-1.0.0-py2.6-linux-x86_64.egg 
    236 }}} 
    237  
    238 And copy the data files into the {{{etc}}} dir: 
    239  
    240 {{{ 
    241 cp -r lib/matplotlib/mpl-data $HOME/ENV/etc/ 
    242 }}} 
    243  
    244 And set data path: 
    245  
    246 {{{ 
    247 export MATPLOTLIBDATA=$HOME/ENV/etc/mpl-data 
    248 }}} 
    249  
    250 Test with: 
    251  
    252 {{{ 
    253 $ python -c 'import matplotlib' 
    254 $ python -c 'from matplotlib.backends.backend_cairo import RendererCairo, FigureCanvasCairo, cairo, Affine2D' 
    255 }}} 
    256  
    257 '''Basemap''' 
    258  
    259 Version: 1.0 
    260  
    261 Using egg: basemap-1.0-py2.6-linux-x86_64.egg 
    262  
    263 Need to extract the egg and build locally, as instructed on wiki page at: 
    264  
    265  http://proj.badc.rl.ac.uk/dcip/wiki/UkcipDdp/Deployment/MatplotlibInstall 
    266  
    267 {{{ 
    268 $ # Environment needs some of these... 
    269 $ export GEOS_DIR=$HOME/external/geos 
    270 $ export LD_LIBRARY_PATH=$HOME/external/geos/lib 
    271 }}} 
    272  
    273 {{{ 
    274 $ # We'll extract an editable version of basemap to ./basemap 
    275 $ cd $HOME/external/src 
    276 $ easy_install -eb . basemap 
    277 $ cd basemap 
    278 $ python setupegg.py bdist_egg 
    279 }}} 
    280  
    281 Now copy the new egg to site-packages and setup basemap-data: 
    282  
    283 {{{ 
    284 $ cp dist/*.egg $HOME/ENV/lib/python2.6/site-packages/ 
    285 $ cp -r lib/mpl_toolkits/basemap/data $HOME/ENV/etc/basemap-data 
    286 $ export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/basemap-1.0-py2.6-linux-x86_64.egg 
    287 $ export BASEMAPDATA=$HOME/ENV/etc/basemap-data 
    288 }}} 
    289  
    290 Environment seemed to then need: 
    291 $ export DBFLIBSO=$HOME/svn/cowsserver/buildout/basemap/build/lib.linux-x86_64-2.6 
    292  
    293 Test with: 
    294  
    295 {{{ 
    296 $ python -c 'import pylab;from mpl_toolkits.basemap import Basemap; b = Basemap();b.drawcoastlines(); pylab.savefig("/tmp/map.png")' 
    297 $ python -c 'from mpl_toolkits import basemap; x =basemap.Basemap(llcrnrlon=-10.5,llcrnrlat=49.5,urcrnrlon=3.5,urcrnrlat=59.5,resolution="i",projection="tmerc",lon_0=-4.36,lat_0=54.7,rsphere=[6370997,1])' 
    298 }}} 
    299  
    300 '''PIL''' 
    301  
    302 Version: 1.1.7 
    303  
    304 pip installed egg: PIL-1.1.7-py2.6-linux-x86_64.egg 
    305  
    306 {{{ 
    307 $ pip install pil==1.1.7 
    308 }}} 
    309  
    310 Test with: 
    311  
    312 {{{ 
    313 $ python -c 'import Image' 
    314 }}} 
    315  
    316 '''Geoplot''' 
    317  
    318 Version: taken from trunk and then edited as follows: 
    319  
    320 {{{ 
    321 $ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/qesdi/geoplot/trunk geoplot 
    322 $ cd geoplot 
    323 }}} 
    324  
    325 Now edit the following line: 
    326  
    327 {{{ 
    328 $ svn diff lib/geoplot/grid_builder_national.py 
    329                                 urcrnrlon=10, urcrnrlat=10, 
    330 +                               rsphere=[6370997,1], 
    331                                 resolution=None) 
    332 }}} 
    333  
    334 Test with: 
    335  
    336 {{{ 
    337 $ python -c 'import geoplot.grid_builder_national' 
    338 }}} 
    339  
    340  
    341 '''image_utils''' 
    342  
    343 Version: 1.0 
    344  
    345 Installed from SVN repository: 
    346  
    347 {{{ 
    348 $ cd $HOME/external 
    349 $ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/qesdi/image_utils/trunk image_utils                      
    350 $ python setup.py develop 
    351 }}} 
    352  
    353 Test with: 
    354  
    355 {{{ 
    356 $ python -c 'import image_utils' 
    357 }}} 
    358  
    359 '''Pylons''' 
    360  
    361 Version: 1.0 
    362  
    363 When you {{{pip install}}} pylons-1.0 you get the following dependencies installed as eggs: 
    364  
    365 {{{ 
    366 Beaker-1.5.4-py2.6.egg 
    367 decorator-3.2.0-py2.6.egg 
    368 FormEncode-1.2.3dev-py2.6.egg 
    369 Mako-0.3.4-py2.6.egg 
    370 MarkupSafe-0.11-py2.6-linux-x86_64.egg 
    371 nose-0.11.4-py2.6.egg 
    372 Paste-1.7.4-py2.6.egg 
    373 PasteDeploy-1.3.4-py2.6.egg 
    374 PasteScript-1.7.3-py2.6.egg 
    375 Routes-1.12.3-py2.6.egg 
    376 simplejson simplejson-2.1.1-py2.6-linux-x86_64.egg 
    377 Tempita-0.5dev-py2.6.egg 
    378 WebError-0.10.2-py2.6.egg 
    379 WebHelpers-1.2-py2.6.egg 
    380 WebOb-1.0-py2.6.egg 
    381 WebTest-1.2.2-py2.6.egg 
    382 }}} 
    383  
    384 So try it: 
    385  
    386 {{{ 
    387 $ pip install pylons==1.0 
    388 }}} 
    389  
    390 '''Mock''' 
    391  
    392 Version: 0.7.0b2 
    393  
    394 Pip installed egg: mock-0.7.0b2-py2.6.egg 
    395  
    396 {{{ 
    397 $ pip install mock==0.7.0b2 
    398 }}} 
    399  
    400 '''Shapely''' 
    401  
    402 Version: 1.2.5 
    403  
    404 Pip installed egg: Shapely-1.2.5-py2.6.egg 
    405  
    406 {{{ 
    407 $ pip install shapely==1.2.5 
    408 }}} 
    409  
    410 '''Genshi''' 
    411  
    412 Version: 0.6 
    413  
    414 Pip installed egg: Genshi-0.6-py2.6.egg 
    415  
    416 {{{ 
    417 $ pip install genshi==0.6 
    418 }}} 
    419  
    420 '''owslib''' 
    421  
    422 Version: 0.3.1 
    423  
    424 pip installed egg: OWSLib-0.3.1-py2.6.egg 
    425  
    426 {{{ 
    427 $ pip install owslib==0.3.1 
    428 }}} 
    429  
    430 cows 
    431 cowsserver 
    432  
    433 === 3. Python packages only used for testing === 
    434  
    435 '''libxml2''' 
    436  
    437 Version: 2.6.21 
    438  
    439 Need to install {{{lib2xmldom}}} which requires {{{libxml2}}}. Need to download source as 
    440 no egg available. 
    441  
    442 {{{ 
    443 $ cd $HOME/external 
    444 $ wget ftp://xmlsoft.org/libxml2/python/libxml2-python-2.6.21.tar.gz 
    445 $ tar xzvf libxml2-python-2.6.21.tar.gz 
    446 $ cd libxml2-python-2.6.21/ 
    447 $ python setup.py build 
    448 $ python setup.py install 
    449 }}} 
    450  
    451 Test with: 
    452  
    453 {{{ 
    454 $ python -c 'import libxml2' 
    455 }}} 
    456  
    457 '''libxml2dom''' 
    458  
    459 Version: 0.4.7 
    460  
    461 pip installed egg: libxml2dom-0.4.7-py2.6.egg 
    462  
    463 {{{ 
    464 pip install libxml2dom==0.4.7 
    465 }}} 
    466  
    467 Test with: 
    468  
    469 {{{ 
    470 $ python -c 'import libxml2dom' 
    471 }}} 
    472  
    473 == Installing cows libraries == 
    474  
    475 At present I've installed from the migrated branch and used develop eggs. 
    476  
    477 COWS: 
    478  
    479 {{{ 
    480 $ cd $HOME/external 
    481 $ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/cows/branches/migrate-py26-pylons10 cows 
    482 $ cd cows 
    483 $ python setup.py develop 
    484 }}} 
    485  
    486 COWS Server: 
    487  
    488 {{{ 
    489 $ cd $HOME/external 
    490 $ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/cowsserver/branches/migrate-py26-pylons10 cowsserver 
    491 $ cd cowsserver 
    492 $ python setup.py develop 
    493 }}} 
    494  
    495 COWS Client: 
    496  
    497 {{{ 
    498 $ cd $HOME/external 
    499 $ svn co http://astephen@proj.badc.rl.ac.uk/svn/ndg/cowsclient/branches/migrate-py26-pylons10 cowsclient 
    500 $ cd cowsclient 
    501 $ python setup.py develop 
    502 }}} 
    503  
    504  
    505 === Setting up the environment to run === 
    506  
    507 The following additions were appended to the {{{activate}}} script in the virtualenv: 
    508  
    509 {{{ 
    510 # Additional environment variables required 
    511 export GEOS_DIR=$HOME/external/geos 
    512 export LD_LIBRARY_PATH=$HOME/external/geos/lib 
    513 export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/matplotlib-1.0.0-py2.6-linux-x86_64.egg 
    514 export MATPLOTLIBDATA=$HOME/ENV/etc/mpl-data 
    515 export PYTHONPATH=$PYTHONPATH:$HOME/ENV/lib/python2.6/site-packages/basemap-1.0-py2.6-linux-x86_64.egg 
    516 export BASEMAPDATA=$HOME/ENV/etc/basemap-data 
    517 }}} 
    518  
    519 Then the environment can be set up with: 
    520  
    521 {{{ 
    522 $ . $HOME/ENV/bin/activate 
    523 }}} 
    524  
    525 == Testing running a service in the above environment == 
    526  
    527 To test the service, refer to the notes on the [wiki:CowsFramework/GettingStarted Getting Started with COWS] page using the 
    528 start directories for COWS server and COWS client apps as: 
    529  
    530 {{{ 
    531 $ $HOME/external/cowsserver 
    532 $ $HOME/external/cowsclient 
    533 }}} 
     1See http://cows.badc.rl.ac.uk/CowsFramework/CowsPython2.6