Changes between Version 19 and Version 20 of CdatLite


Ignore:
Timestamp:
15/09/09 16:17:28 (10 years ago)
Author:
spascoe
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • CdatLite

    v19 v20  
    1  
    2 = Sub-Pages of this page = 
    3  
    4 [[TitleIndex(CdatLite/)]] 
    5  
    6 [[PageOutline]] 
    7  
    8  
    9  
    10 = cdat_lite-5.0 = 
    11  
    12 cdat_lite-5.0 currently mirrors the 5.0beta CDAT codebase.  Watch this space for upgrade to CDAT-5.0 full release. 
    13  
    14 Version 5.0 of cdat_lite introduces many changes. 
    15  
    16  1. It is built directly from the CDAT-5.0 codebase and version numbers are completely synchronised.  CDAT-5.0  
    17     requires Python-2.5 and uses {{{numpy}}} rather than {{{Numeric}}}.  See [#Upgrading Upgrading] for what you need  
    18     to do to your code. 
    19  1. It is now only distributed as a tarball.  Binary distributions tended to run into library version conflicts 
    20     on different versions of linux.  However {{{easy_install}}} should still be able to compile the package 
    21     automatically.  See [#Installing Installing].  Also you can make your own binary eggs for given OSes. 
    22  1. You now need to have the netcdf library pre-installed.  Automatically building netcdf wasn't portable enough. 
    23  
    24 == Installing == 
    25  
    26 First satisfy a couple of dependencies.   
    27  
    28 Make sure you have the [http://www.unidata.ucar.edu/software/netcdf/ netcdf libraries] installed.  If they are in a non-standard location make a note of it (you will be prompted) or set {{{NETCDF_HOME}}} to its installation prefix.  NB, if you are running on a 64bit machine, be aware that netcdf may have installed in the 32bit directories - e.g. /usr/local/lib.  The cdat setup script may check for the files in the 64bit directories so may fail.  Either ensure the install is available in the 64bit directories or link the files - NB, two files are required: /usr/local/lib/libnetcdf.a and /usr/local/include/netcdf.h. 
    29  
    30 Install {{{numpy}}}.  If you aren't bothered about linear algebra optimisation this usually works with {{{easy_install}}}.  E.g. 
    31  
    321{{{ 
    33 $ easy_install numpy 
     2#!rst 
     3========= 
     4CDAT-Lite 
     5========= 
     6 
     7.. contents:: 
     8 
     9.. sectnum:: 
     10 
     11CDAT-Lite is a Python package for managing and analysing climate 
     12science data.  It is a subset of the Climate Data Analysis Tools 
     13(CDAT_) developed by PCMDI_ at Lawrence Livermore National Laboratory. 
     14 
     15CDAT-lite aims to compliment CDAT by focussing on it's core data 
     16management and analysis components and by offering a radically 
     17different installation system to CDAT.  As a result it is much more 
     18lightweight (hence the name): CDAT's source distribution is the order 
     19of 1Gb whereas CDAT-lite is under 5Mb. 
     20 
     21.. _CDAT: http://www2-pcmdi.llnl.gov/cdat 
     22.. _PCMDI: http://www2-pcmdi.llnl.gov/ 
     23 
     24FAQ 
     25=== 
     26 
     27What is CDAT? 
     28------------- 
     29 
     30CDAT_ is a large suite of open source tools distributed by PCMDI_ for 
     31the management and analysis of climate data.  It includes several 
     32visualisation components and the graphical user interface VCDAT. 
     33 
     34What is the difference between CDAT and cdat-lite? 
     35-------------------------------------------------- 
     36 
     37Differences between CDAT and CDAT-lite can be classified as 
     38differences in scope, i.e. which packages are included, and installation system. 
     39 
     40cdat-lite contains the 'cdms2' package and a few related 
     41packages.  It does not include the 'vcs' visualisation package or the 
     42VCDAT graphical user interface.  As of v5.1.1-0.3pre3 the included 
     43packages are: 
     44 
     45 * cdms2 
     46 
     47 * cdtime 
     48 
     49 * cdutil 
     50 
     51 * genutil 
     52 
     53 * ncml 
     54 
     55 * Properties 
     56 
     57 * regrid2 
     58 
     59 * unidataa 
     60 
     61 * xmgrace 
     62 
     63CDAT bundles virtually all dependencies together in it's source 
     64distribution -- even Python itself.  This has it's advantages as it 
     65simplifies satisfying dependencies and avoids version conflicts 
     66between dependencies.  However, if you want to integrate CDAT's data 
     67management components into your existing Python architecture CDAT can 
     68be overkill. 
     69 
     70 
     71What has changed between cdat-lite-4.x and cdat-lite-5.x? 
     72--------------------------------------------------------- 
     73 
     74If you are a cdat-lite-4 user (or a CDAT 4 user) you have a big 
     75migration job on your hands.  CDAT-4 uses the ``Numeric`` package for 
     76arrays which has been out of date and unmaintained for a long time 
     77now.  It is known to have problems on 64bit architectures. 
     78 
     79How does cdat-lite track changes to CDAT? 
     80----------------------------------------- 
     81 
     82cdat-lite tries to release major new versions shortly after new 
     83versions of CDAT.  Sometimes CDAT-trunk contains important fixes that 
     84should be applied so that the latest cdat_lite can run ahead of 
     85official CDAT releases (although sometimes CDAT recommends you build 
     86from trunk anyway). 
     87 
     88The one exception is the UK Met. Office PP file support which is 
     89usually updated in cdat_lite before CDAT.  In all cases the exact 
     90build versions of CDAT and cdunifpp will be stated in the 
     91distribution's ``setup.py`` file. 
     92 
     93How can I use CMOR2 with cdat-lite? 
     94----------------------------------- 
     95 
     96We are interested to hear any with experience of using CMOR2 with 
     97cdat-lite but it should be as simple as downloading the distribution 
     98and installing it in parallel with:: 
     99 
     100  # From the CMOR install directory 
     101  $ python setup.py install 
     102 
     103How can I use OPeNDAP with cdat-lite? 
     104------------------------------------- 
     105 
     106OPeNDAP support is an experimental feature of cdat-lite at the moment. 
     107Unlike CDAT you don't select OPeNDAP explicitly during installation 
     108but cdat-lite will inherit any OPeNDAP support embedded into the 
     109NetCDF4 library.  Recent beta releases of NetCDF4 provides a switch to 
     110transparently use OPeNDAP. 
     111 
     112How do I install cdat-lite as an unprivileged user? 
     113--------------------------------------------------- 
     114 
     115See `installing locally`_ 
     116 
     117Which versions of NetCDF does cdat-lite support? 
     118------------------------------------------------ 
     119 
     120TODO 
     121 
     122Installing cdat-lite 
     123==================== 
     124 
     125cdat-lite is distributed as a tarball available from the `cdat-lite 
     126homepage`_ on the `NERC Data Grid wiki` .  It is also installable 
     127using the ``easy_install`` tool.  If you are familiar with 
     128``easy_install`` try this super-quick installation recipe:: 
     129 
     130  $ export NETCDF_HOME=/usr/local/netcdf 
     131  # Required if using a NetCDF4 compiled with HDF5 
     132  $ export HDF5_HOME=/usr/local/hdf5 
     133  $ easy_install cdat_lite 
     134 
     135 
     136Dependencies 
     137------------ 
     138 
     139To install cdat-lite you will need: 
     140 
     141 1. `Python 2.5.x`_.  cdat-lite has not been tested on 2.6 but may 
     142 work (feedback would be gratefully received).  It is unlikely to work on 3.0. 
     143 
     144 2. `setuptools`_.  cdat-lite will attempt to download and install 
     145 setuptools if it is missing but it is safer to install it first. 
     146 
     147 3. `NetCDF-3.x`_ or greater.  cdat-lite should work with any 
     148 relatively modern NetCDF3 installation on your system provided it is 
     149 compiled as a shared library.  It will also work with NetCDF4 
     150 configured in various different ways, including embedded OPeNDAP 
     151 mode. 
     152 
     153 4. If you want to run the short test suite you will need nose_ 
     154 
     155.. _`Python 2.5.x`: http://www.python.org/download/releases/2.5.4 
     156.. _`setuptools`: http://pypi.python/org/setuptools 
     157.. _`NetCDF-3.x`: http://www.unidata.ucar.edu/software/netcdf/ 
     158.. _nose: http://somethingaboutorange.com/mrl/projects/nose/ 
     159 
     160Selecting your NetCDF installation 
     161---------------------------------- 
     162 
     163cdat-lite will work with NetCDF3 or NetCDF4 but because it is 
     164referenced by shared libraries (the python C extension modules) it 
     165must be compiled as position independent code.  It is probably easiest 
     166to install NetCDF as a shared library (use ``--enable-shared`` in the 
     167NetCDF ``configure`` script).  Alternatively, you can configure NetCDF with:: 
     168 
     169  $ ./configure --with-pic ... 
     170 
     171cdat-lite will look for a NetCDF installation in several places and 
     172prompt you if it can't find it.  If your NetCDF is installed somewhere 
     173unusual, or if you want to select a specific installation, set the 
     174NETCDF_HOME variable.  E.g.:: 
     175 
     176  # sh users 
     177  $ export NETCDF_HOME=/usr/local/netcdf 
     178  # csh users 
     179  $ setenv NETCDF_HOME /usr/local/netcdf 
     180 
     181If you are using NetCDF4 cdat-lite will also look for your HDF5 
     182installation which you can configure in a similar way:: 
     183 
     184  # sh users 
     185  $ export HDF5_HOME=/usr/local/hdf5 
     186  # csh users 
     187  $ setenv HDF5_HOME /usr/local/hdf5 
     188 
     189Note, you don't need these environment variables set to run cdat_lite, 
     190although the libraries must be findable by your system's dynamic 
     191linker.  This can be configured by setting ``LD_LIBRARY_PATH`` or using ``ldconfig``. 
     192 
     193Running the installer 
     194--------------------- 
     195 
     196If you have all the dependencies in place you can try using 
     197``easy_install`` to automatically download and install cdat_lite.  Make sure you have access to the internet, with the appropriate HTTP proxy settings, and do:: 
     198 
     199  $ easy_install cdat-lite 
     200 
     201Alternatively you might want to see what you are installing :-).  In 
     202this case either download the tarball__ or use ``easy_install`` to do it for you:: 
     203 
     204  $ easy_install -eb . cdat-lite 
     205  # The cdat-lite tarball will be downloaded unpacked into you current directory 
     206 
     207Now from the distribution directory run the build and install steps separately:: 
     208 
     209  $ python setup.py bdist_egg 
     210  $ easy_install dist/cdat-lite*.egg 
     211 
     212__ `cdat-lite homepage`_ 
     213 
     214 
     215.. _`installing locally`: 
     216 
     217Installing as an unprivileged user 
     218---------------------------------- 
     219 
     220If you don't have write access to your python distribution you can use 
     221the tool virtualenv_ to create a local python environment with 
     222it's own ``easy_install`` executable which you can then use to install 
     223cdat-lite.  In combination with ``NETCDF_HOME``, ``HDF5_HOME`` and 
     224``LD_LIBRARY_PATH`` it should be possible to install all dependencies 
     225of cdat-lite locally.  See the virtualenv_ for details on 
     226installation or try this recipe after downloading the virtualenv:: 
     227 
     228  # From virtualenv distribution directory 
     229  $ ./virtualenv.py <virtualenv-path> 
     230  $ cd <virtualenv-path> 
     231  $ source bin/activate 
     232  (venv)$ easy_install cdat-lite 
     233 
     234.. _virtualenv: http://pypi.python.org/pypi/virtualenv 
     235 
     236Testing the installation 
     237======================== 
     238 
     239cdat-lite ships with a small set of tests designed to verify that it 
     240has been built successfuly.  These tests require the testing framework 
     241nose_.  Once cdat-lite is installed just run:: 
     242 
     243  $ nosetests cdat_lite 
     244 
     245When run from cdat-lite's distribution directory nosetests will run 
     246slightly differently, running some tests that are known to fail at the 
     247moment.  To disable this behaviour do: 
     248 
     249  $ nosetests --config='' 
     250 
     251.. _`cdat-lite homepage`: http://proj.badc.rl.ac.uk/ndg/wiki/CdatLite 
     252.. _`NERC Data Grid wiki`: http://proj.badc.rl.ac.uk/ndg/wiki 
    34253}}} 
    35  
    36 Now you are ready to install {{{cdat_lite}}}.  On many linux systems the C compiler is incorrectly detected therefore 
    37 it is safest to do: 
    38  
    39 {{{ 
    40 $ CC=gcc easy_install cdat_lite 
    41 }}} 
    42  
    43 === Advanced Installation === 
    44  
    45 You might want more control over the installation process but you can still get easy_install to download the tarball 
    46 for you: 
    47  
    48 {{{ 
    49 $ easy_install -eb <dir> cdat_lite 
    50 }}} 
    51  
    52 This will download and unpack the tarball into {{{<dir>}}} then  you can run the {{{setup.py}}} script directly or build a binary egg for installation elsewhere. 
    53  
    54 == Upgrading == 
    55  
    56 The move from Python-2.4 to Python-2.5 and from Numeric to numpy requires very few changes to your code.  Most changes that are needed can be done 
    57 automatically using the {{{convertcdms}}} script. 
    58  
    59 {{{ 
    60 $ convertcdms 
    61 Usage: 
    62     convertcdms.py [options] path 
    63  
    64       Convert scripts that use the Numeric, MA, cdms, and regrid modules to numpy, ma, cdms2, and regrid2. 
    65  
    66     -or- 
    67  
    68     convertcdms.py [options] -r direc 
    69  
    70       Convert all Numeric/MA/cdms Python scripts and C source code in directory 'direc' to numpy/ma/cdms2. 
    71  
    72       Use this form to convert C source code. 
    73  
    74 Arguments: 
    75  
    76     path: A .py file to convert 
    77     direc: A directory: convert all .py, .c, and .h files 
    78  
    79 Options: 
    80  
    81     -a:           Disable 'aggressive' conversion (see Note 1) 
    82     -c:           Clobber the original file(s). The default is to move the original to <name>.orig 
    83     -h:           Print a help message 
    84     -n:           Do not write '# Adapted by...' comment at the top of file. 
    85     -r:           Recursively convert all .py and .c/.h files in directory direc 
    86     -s direc:     Skip the directory. This option may be used more than once. 
    87                     For example, '-s .svn' skips all subversion subdirectories. 
    88  
    89 Notes: 
    90  
    91     1) The -a option turns off the following translations: 
    92  
    93         - MA.Float => Numeric.Float, similarly for MA.Int, MA.NewAxis 
    94         - MA.Numeric => Numeric 
    95         - The 'typecode=' argument in MA functions is changed to 'dtype=' 
    96         - XX.mask() => XX.mask 
    97         - XX.mask is None => ((XX.mask is None) or (XX.mask is MV2.nomask)) 
    98         - A keyword argument 'axis=0' is added to MA.sum, MA.average, MA.product, and MA.repeat 
    99  
    100     2) By default the following translations are made: 
    101  
    102         - import cdms => import cdms2 as cdms 
    103         - import regrid => import regrid2 as regrid 
    104         - import MV => import MV2 as MV 
    105         - import Numeric => import numpy.oldnumeric as Numeric 
    106         - import MA => import numpy.oldnumeric.ma as MA 
    107         - import cdms.MV => import cdms2.MV2 
    108         - from cdms import XX => from cdms2 import XX (similarly for regrid, MV, MA, and Numeric) 
    109         - from cdms.XX import YY => from cdms2.XX import YY (similarly for regrid, MV, MA, and Numeric) 
    110         - import cdms as XX => import cdms2 as XX (similarly for regrid, MV, MA, and Numeric) 
    111         - import cdms.XX => import cdms2.XX (similarly for regrid, MV, MA, and Numeric) 
    112         - import XX, cdms, YY => import XX, cdms2 as cdms, YY (similarly for regrid, MV, MA, and Numeric) 
    113         - the translations in Note (1) 
    114         - the translations in numpy.oldnumeric.alter_code1. This module is used for most Numeric and MA-related translations. 
    115 }}} 
    116  
    117 A few minor changes may also be required.  If you use arrays in logical expressions you may need to change your code.  The following would work with Numeric: 
    118 {{{ 
    119 >>> import numpy 
    120 >>> x = numpy.arange(4) 
    121 >>> if x: 
    122 ...   print 'hello' 
    123 ... 
    124 Traceback (most recent call last): 
    125   File "<stdin>", line 1, in <module> 
    126 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() 
    127 >>>                                        
    128 }}} 
    129  
    130  
    131 == Installation problems (and solutions) == 
    132 On a 64-bit machine you may get a message like: 
    133 {{{ 
    134 `a local symbol' can not be used when making a shared object; recompile with -fPIC 
    135 /usr/lib/gcc/x86_64-linux-gnu/3.4.6/../../../../lib/libnetcdf.a: could not read symbols: Bad value 
    136 }}} 
    137  
    138 In which case you need to recompile your NetCDF installation to use the fPIC option as follows: 
    139  
    140 {{{ 
    141 ./configure --with-pic 
    142 make 
    143 make install 
    144 }}} 
    145  
    146 = CDAT-lite 4.3 = 
    147  
    148 cdat-lite is a simple repackaging of the I/O layer of the [http://www-pcmdi.llnl.gov/software-portal/cdat: Climate Data Analysis Tools (CDAT)] as a [wiki:PythonEggs: Python Egg].  A short article on the reasons behind cdat-lite and some of the design decisions made can be found [http://www.lirico.co.uk/wp/?p=20 here] on [http://www.lirico.co.uk Stephen Pascoe's blog]. 
    149  
    150 == Features == 
    151  
    152  * Includes cdscan from the comand-line or from within python 
    153    using the cdat_lite.scripts.cdscan module. 
    154  * Includes the latest version of the British Atmospheric Data 
    155    Centre's cdunifpp module which provides read access to UK 
    156    Met. Office PP files. 
    157  * Downloads and installs Numeric on demand.  If you have Numeric installed 
    158    already but the package doesn't have an egg-info directory easy-install will 
    159    re-install Numeric for you unless you use the --no-deps switch. 
    160  * Currently focused on providing NetCDF and PP format support.  DRS, HDF and DODS (OpeNDAP) 
    161    are disabled in the default egg build. 
    162  
    163 == Package Overview == 
    164  
    165  * CDAT core packages:  
    166    * cdms, cdtime, cdutil, genutil, Properties, regrid, unidata, xmgrace 
    167    These top-level packages can be used in the same way you would with a full CDAT installation.  E.g. 
    168    {{{ 
    169    #!python 
    170    >>> import cdms 
    171    >>> f = cdms.open('dataset.nc') 
    172    }}} 
    173  * cdat_lite support packages: These packages support the cdat_lite 
    174    egg installation. 
    175    * cdat_lite.clib: libraries and headers used to build CDAT source: 
    176      libcdms and libnetcdf.   
    177    * cdat_lite.scripts: contains a version of cdscan suitable for declaring as 
    178      a setuptools entry-point.  During installation 
    179      easy-install will create a stub script to execute cdscan on the 
    180      command line 
    181    * cdat_lite.test: contains a test suite. 
    182  
    183 == Installation == 
    184  
    185 Cdat-lite uses [http://peak.telecommunity.com/DevCenter/EasyInstall easy_install] as its installation mechanism.  This makes downloading and installing the software a one-step process, once you've installed the [http://peak.telecommunity.com/DevCenter/setuptools setuptools] package (itself very easy to install).   
    186  
    187 Binary eggs and the source tarball of {{{cdat_lite}}} are available from [http://ndg.nerc.ac.uk/dist].  It is also registered at the [http://cheeseshop.python.org/pypi Python Cheeseshop] so should be discovered automatically by {{{easy_install}}}. 
    188  
    189  1. Start with a python 2.4+ installation with distutils.  You cannot build or run the egg without distutils.  If your python has been installed by a unix package manager then you may have to install the python-devel package.  To check you've got distutils type the following at the Python prompt: 
    190  {{{ 
    191  >>> import distutils 
    192  >>> 
    193  }}} 
    194  1. Install [http://peak.telecommunity.com/DevCenter/setuptools setuptools].  This includes the {{{easy_install}}} script.  The setuptools website has detailed installation instructions but if you have write-permission to your python installation the following should work: 
    195  {{{ 
    196  $ wget http://peak.telecommunity.com/dist/ez_setup.py 
    197  $ python ez_setup.py 
    198  }}} 
    199  1. Now you're ready to install cdat_lite.  Simply type: 
    200  {{{ 
    201  $ easy_install cdat_lite 
    202  }}} 
    203  1. Alternatively, if you want to install into a local directory {{{$EGG_DIR}}}, ensure {{{$EGG_DIR}}} exists and is on your {{{$PYTHONPATH}}} then type: 
    204  {{{ 
    205  $ easy_install -d $EGG_DIR cdat_lite 
    206  }}} 
    207  1. For more installation options see the [http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations easy_install documentation] 
    208  
    209  
    210 == Versioning == 
    211  
    212 cdat-lite versioning is complicated as it is a repackage of CDAT with 
    213 an updated cdunifpp module. There are therefore 3 version numbers to 
    214 consider: CDAT, cdunifpp and cdat-lite.  Once we start using versions 
    215 of CDAT from the PCMDI SVN the version string could become very long 
    216 indeed therefore the following strategy is used: 
    217  
    218  1. Eggs are quoted with the version: <cdat-release>-<cdat-lite-version> 
    219  1. If a PCMDI SVN version of CDAT is used it is stated in long_description not in <cdat-release>. 
    220  1. The cdunifpp version is stated in long_description not in the version.  Any change to the cdunifpp version naturally triggers a new <cdat-lite-version>. 
    221  
    222  
    223  
    224 == Using CDMS scripts == 
    225  
    226 cdat_lite includes the libcdms script cdscan within the egg.  You have 
    227 several options to get at it. 
    228  
    229 === The stub script === 
    230  
    231 When you install the egg with easy_install a script stub is created 
    232 automatically which calls cdscan within the egg.  Therefore you should 
    233 be able to use cdscan as usual.  If you've installed the egg in a 
    234 local directory the script stub will be in that directory. 
    235  
    236 === Import cdscan as a module from within python === 
    237 {{{ 
    238 #!python  
    239 >>> import cdat_scripts.cdscan as cdscan 
    240 >>> cdscan.main(['cdscan', '-x', 'dataset.xml', filename, ...]) 
    241 }}} 
    242  
    243 The argument to main() must be the equivilent of sys.argv when run 
    244 from the command line, i.e. the first element should be a pseudo 
    245 command name. 
    246  
    247 === Import a cdscan entry point using the pkg_resources API. === 
    248  
    249 This is the way scripts are supposed to be found from within eggs and is 
    250 used by easy_install to automatically create script stubs. 
    251 {{{ 
    252 #!python 
    253 >>> import pkg_resources 
    254 >>> cdscan = pkg_resources.load_entry_point('cdat_lite', 'console_scripts', 'cdscan') 
    255 >>> cdscan() 
    256 }}} 
    257 In this case sys.argv must be set to the arguments you want. 
    258