Changes between Version 19 and Version 20 of CdatLite

15/09/09 16:17:28 (12 years ago)



  • CdatLite

    v19 v20  
    2 = Sub-Pages of this page = 
    4 [[TitleIndex(CdatLite/)]] 
    6 [[PageOutline]] 
    10 = cdat_lite-5.0 = 
    12 cdat_lite-5.0 currently mirrors the 5.0beta CDAT codebase.  Watch this space for upgrade to CDAT-5.0 full release. 
    14 Version 5.0 of cdat_lite introduces many changes. 
    16  1. It is built directly from the CDAT-5.0 codebase and version numbers are completely synchronised.  CDAT-5.0  
    17     requires Python-2.5 and uses {{{numpy}}} rather than {{{Numeric}}}.  See [#Upgrading Upgrading] for what you need  
    18     to do to your code. 
    19  1. It is now only distributed as a tarball.  Binary distributions tended to run into library version conflicts 
    20     on different versions of linux.  However {{{easy_install}}} should still be able to compile the package 
    21     automatically.  See [#Installing Installing].  Also you can make your own binary eggs for given OSes. 
    22  1. You now need to have the netcdf library pre-installed.  Automatically building netcdf wasn't portable enough. 
    24 == Installing == 
    26 First satisfy a couple of dependencies.   
    28 Make sure you have the [ netcdf libraries] installed.  If they are in a non-standard location make a note of it (you will be prompted) or set {{{NETCDF_HOME}}} to its installation prefix.  NB, if you are running on a 64bit machine, be aware that netcdf may have installed in the 32bit directories - e.g. /usr/local/lib.  The cdat setup script may check for the files in the 64bit directories so may fail.  Either ensure the install is available in the 64bit directories or link the files - NB, two files are required: /usr/local/lib/libnetcdf.a and /usr/local/include/netcdf.h. 
    30 Install {{{numpy}}}.  If you aren't bothered about linear algebra optimisation this usually works with {{{easy_install}}}.  E.g. 
    33 $ easy_install numpy 
     7.. contents:: 
     9.. sectnum:: 
     11CDAT-Lite is a Python package for managing and analysing climate 
     12science data.  It is a subset of the Climate Data Analysis Tools 
     13(CDAT_) developed by PCMDI_ at Lawrence Livermore National Laboratory. 
     15CDAT-lite aims to compliment CDAT by focussing on it's core data 
     16management and analysis components and by offering a radically 
     17different installation system to CDAT.  As a result it is much more 
     18lightweight (hence the name): CDAT's source distribution is the order 
     19of 1Gb whereas CDAT-lite is under 5Mb. 
     21.. _CDAT: 
     22.. _PCMDI: 
     27What is CDAT? 
     30CDAT_ is a large suite of open source tools distributed by PCMDI_ for 
     31the management and analysis of climate data.  It includes several 
     32visualisation components and the graphical user interface VCDAT. 
     34What is the difference between CDAT and cdat-lite? 
     37Differences between CDAT and CDAT-lite can be classified as 
     38differences in scope, i.e. which packages are included, and installation system. 
     40cdat-lite contains the 'cdms2' package and a few related 
     41packages.  It does not include the 'vcs' visualisation package or the 
     42VCDAT graphical user interface.  As of v5.1.1-0.3pre3 the included 
     43packages are: 
     45 * cdms2 
     47 * cdtime 
     49 * cdutil 
     51 * genutil 
     53 * ncml 
     55 * Properties 
     57 * regrid2 
     59 * unidataa 
     61 * xmgrace 
     63CDAT bundles virtually all dependencies together in it's source 
     64distribution -- even Python itself.  This has it's advantages as it 
     65simplifies satisfying dependencies and avoids version conflicts 
     66between dependencies.  However, if you want to integrate CDAT's data 
     67management components into your existing Python architecture CDAT can 
     68be overkill. 
     71What has changed between cdat-lite-4.x and cdat-lite-5.x? 
     74If you are a cdat-lite-4 user (or a CDAT 4 user) you have a big 
     75migration job on your hands.  CDAT-4 uses the ``Numeric`` package for 
     76arrays which has been out of date and unmaintained for a long time 
     77now.  It is known to have problems on 64bit architectures. 
     79How does cdat-lite track changes to CDAT? 
     82cdat-lite tries to release major new versions shortly after new 
     83versions of CDAT.  Sometimes CDAT-trunk contains important fixes that 
     84should be applied so that the latest cdat_lite can run ahead of 
     85official CDAT releases (although sometimes CDAT recommends you build 
     86from trunk anyway). 
     88The one exception is the UK Met. Office PP file support which is 
     89usually updated in cdat_lite before CDAT.  In all cases the exact 
     90build versions of CDAT and cdunifpp will be stated in the 
     91distribution's ```` file. 
     93How can I use CMOR2 with cdat-lite? 
     96We are interested to hear any with experience of using CMOR2 with 
     97cdat-lite but it should be as simple as downloading the distribution 
     98and installing it in parallel with:: 
     100  # From the CMOR install directory 
     101  $ python install 
     103How can I use OPeNDAP with cdat-lite? 
     106OPeNDAP support is an experimental feature of cdat-lite at the moment. 
     107Unlike CDAT you don't select OPeNDAP explicitly during installation 
     108but cdat-lite will inherit any OPeNDAP support embedded into the 
     109NetCDF4 library.  Recent beta releases of NetCDF4 provides a switch to 
     110transparently use OPeNDAP. 
     112How do I install cdat-lite as an unprivileged user? 
     115See `installing locally`_ 
     117Which versions of NetCDF does cdat-lite support? 
     122Installing cdat-lite 
     125cdat-lite is distributed as a tarball available from the `cdat-lite 
     126homepage`_ on the `NERC Data Grid wiki` .  It is also installable 
     127using the ``easy_install`` tool.  If you are familiar with 
     128``easy_install`` try this super-quick installation recipe:: 
     130  $ export NETCDF_HOME=/usr/local/netcdf 
     131  # Required if using a NetCDF4 compiled with HDF5 
     132  $ export HDF5_HOME=/usr/local/hdf5 
     133  $ easy_install cdat_lite 
     139To install cdat-lite you will need: 
     141 1. `Python 2.5.x`_.  cdat-lite has not been tested on 2.6 but may 
     142 work (feedback would be gratefully received).  It is unlikely to work on 3.0. 
     144 2. `setuptools`_.  cdat-lite will attempt to download and install 
     145 setuptools if it is missing but it is safer to install it first. 
     147 3. `NetCDF-3.x`_ or greater.  cdat-lite should work with any 
     148 relatively modern NetCDF3 installation on your system provided it is 
     149 compiled as a shared library.  It will also work with NetCDF4 
     150 configured in various different ways, including embedded OPeNDAP 
     151 mode. 
     153 4. If you want to run the short test suite you will need nose_ 
     155.. _`Python 2.5.x`: 
     156.. _`setuptools`: http://pypi.python/org/setuptools 
     157.. _`NetCDF-3.x`: 
     158.. _nose: 
     160Selecting your NetCDF installation 
     163cdat-lite will work with NetCDF3 or NetCDF4 but because it is 
     164referenced by shared libraries (the python C extension modules) it 
     165must be compiled as position independent code.  It is probably easiest 
     166to install NetCDF as a shared library (use ``--enable-shared`` in the 
     167NetCDF ``configure`` script).  Alternatively, you can configure NetCDF with:: 
     169  $ ./configure --with-pic ... 
     171cdat-lite will look for a NetCDF installation in several places and 
     172prompt you if it can't find it.  If your NetCDF is installed somewhere 
     173unusual, or if you want to select a specific installation, set the 
     174NETCDF_HOME variable.  E.g.:: 
     176  # sh users 
     177  $ export NETCDF_HOME=/usr/local/netcdf 
     178  # csh users 
     179  $ setenv NETCDF_HOME /usr/local/netcdf 
     181If you are using NetCDF4 cdat-lite will also look for your HDF5 
     182installation which you can configure in a similar way:: 
     184  # sh users 
     185  $ export HDF5_HOME=/usr/local/hdf5 
     186  # csh users 
     187  $ setenv HDF5_HOME /usr/local/hdf5 
     189Note, you don't need these environment variables set to run cdat_lite, 
     190although the libraries must be findable by your system's dynamic 
     191linker.  This can be configured by setting ``LD_LIBRARY_PATH`` or using ``ldconfig``. 
     193Running the installer 
     196If you have all the dependencies in place you can try using 
     197``easy_install`` to automatically download and install cdat_lite.  Make sure you have access to the internet, with the appropriate HTTP proxy settings, and do:: 
     199  $ easy_install cdat-lite 
     201Alternatively you might want to see what you are installing :-).  In 
     202this case either download the tarball__ or use ``easy_install`` to do it for you:: 
     204  $ easy_install -eb . cdat-lite 
     205  # The cdat-lite tarball will be downloaded unpacked into you current directory 
     207Now from the distribution directory run the build and install steps separately:: 
     209  $ python bdist_egg 
     210  $ easy_install dist/cdat-lite*.egg 
     212__ `cdat-lite homepage`_ 
     215.. _`installing locally`: 
     217Installing as an unprivileged user 
     220If you don't have write access to your python distribution you can use 
     221the tool virtualenv_ to create a local python environment with 
     222it's own ``easy_install`` executable which you can then use to install 
     223cdat-lite.  In combination with ``NETCDF_HOME``, ``HDF5_HOME`` and 
     224``LD_LIBRARY_PATH`` it should be possible to install all dependencies 
     225of cdat-lite locally.  See the virtualenv_ for details on 
     226installation or try this recipe after downloading the virtualenv:: 
     228  # From virtualenv distribution directory 
     229  $ ./ <virtualenv-path> 
     230  $ cd <virtualenv-path> 
     231  $ source bin/activate 
     232  (venv)$ easy_install cdat-lite 
     234.. _virtualenv: 
     236Testing the installation 
     239cdat-lite ships with a small set of tests designed to verify that it 
     240has been built successfuly.  These tests require the testing framework 
     241nose_.  Once cdat-lite is installed just run:: 
     243  $ nosetests cdat_lite 
     245When run from cdat-lite's distribution directory nosetests will run 
     246slightly differently, running some tests that are known to fail at the 
     247moment.  To disable this behaviour do: 
     249  $ nosetests --config='' 
     251.. _`cdat-lite homepage`: 
     252.. _`NERC Data Grid wiki`: 
    36 Now you are ready to install {{{cdat_lite}}}.  On many linux systems the C compiler is incorrectly detected therefore 
    37 it is safest to do: 
    39 {{{ 
    40 $ CC=gcc easy_install cdat_lite 
    41 }}} 
    43 === Advanced Installation === 
    45 You might want more control over the installation process but you can still get easy_install to download the tarball 
    46 for you: 
    48 {{{ 
    49 $ easy_install -eb <dir> cdat_lite 
    50 }}} 
    52 This will download and unpack the tarball into {{{<dir>}}} then  you can run the {{{}}} script directly or build a binary egg for installation elsewhere. 
    54 == Upgrading == 
    56 The move from Python-2.4 to Python-2.5 and from Numeric to numpy requires very few changes to your code.  Most changes that are needed can be done 
    57 automatically using the {{{convertcdms}}} script. 
    59 {{{ 
    60 $ convertcdms 
    61 Usage: 
    62 [options] path 
    64       Convert scripts that use the Numeric, MA, cdms, and regrid modules to numpy, ma, cdms2, and regrid2. 
    66     -or- 
    68 [options] -r direc 
    70       Convert all Numeric/MA/cdms Python scripts and C source code in directory 'direc' to numpy/ma/cdms2. 
    72       Use this form to convert C source code. 
    74 Arguments: 
    76     path: A .py file to convert 
    77     direc: A directory: convert all .py, .c, and .h files 
    79 Options: 
    81     -a:           Disable 'aggressive' conversion (see Note 1) 
    82     -c:           Clobber the original file(s). The default is to move the original to <name>.orig 
    83     -h:           Print a help message 
    84     -n:           Do not write '# Adapted by...' comment at the top of file. 
    85     -r:           Recursively convert all .py and .c/.h files in directory direc 
    86     -s direc:     Skip the directory. This option may be used more than once. 
    87                     For example, '-s .svn' skips all subversion subdirectories. 
    89 Notes: 
    91     1) The -a option turns off the following translations: 
    93         - MA.Float => Numeric.Float, similarly for MA.Int, MA.NewAxis 
    94         - MA.Numeric => Numeric 
    95         - The 'typecode=' argument in MA functions is changed to 'dtype=' 
    96         - XX.mask() => XX.mask 
    97         - XX.mask is None => ((XX.mask is None) or (XX.mask is MV2.nomask)) 
    98         - A keyword argument 'axis=0' is added to MA.sum, MA.average, MA.product, and MA.repeat 
    100     2) By default the following translations are made: 
    102         - import cdms => import cdms2 as cdms 
    103         - import regrid => import regrid2 as regrid 
    104         - import MV => import MV2 as MV 
    105         - import Numeric => import numpy.oldnumeric as Numeric 
    106         - import MA => import as MA 
    107         - import cdms.MV => import cdms2.MV2 
    108         - from cdms import XX => from cdms2 import XX (similarly for regrid, MV, MA, and Numeric) 
    109         - from cdms.XX import YY => from cdms2.XX import YY (similarly for regrid, MV, MA, and Numeric) 
    110         - import cdms as XX => import cdms2 as XX (similarly for regrid, MV, MA, and Numeric) 
    111         - import cdms.XX => import cdms2.XX (similarly for regrid, MV, MA, and Numeric) 
    112         - import XX, cdms, YY => import XX, cdms2 as cdms, YY (similarly for regrid, MV, MA, and Numeric) 
    113         - the translations in Note (1) 
    114         - the translations in numpy.oldnumeric.alter_code1. This module is used for most Numeric and MA-related translations. 
    115 }}} 
    117 A few minor changes may also be required.  If you use arrays in logical expressions you may need to change your code.  The following would work with Numeric: 
    118 {{{ 
    119 >>> import numpy 
    120 >>> x = numpy.arange(4) 
    121 >>> if x: 
    122 ...   print 'hello' 
    123 ... 
    124 Traceback (most recent call last): 
    125   File "<stdin>", line 1, in <module> 
    126 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() 
    127 >>>                                        
    128 }}} 
    131 == Installation problems (and solutions) == 
    132 On a 64-bit machine you may get a message like: 
    133 {{{ 
    134 `a local symbol' can not be used when making a shared object; recompile with -fPIC 
    135 /usr/lib/gcc/x86_64-linux-gnu/3.4.6/../../../../lib/libnetcdf.a: could not read symbols: Bad value 
    136 }}} 
    138 In which case you need to recompile your NetCDF installation to use the fPIC option as follows: 
    140 {{{ 
    141 ./configure --with-pic 
    142 make 
    143 make install 
    144 }}} 
    146 = CDAT-lite 4.3 = 
    148 cdat-lite is a simple repackaging of the I/O layer of the [ Climate Data Analysis Tools (CDAT)] as a [wiki:PythonEggs: Python Egg].  A short article on the reasons behind cdat-lite and some of the design decisions made can be found [ here] on [ Stephen Pascoe's blog]. 
    150 == Features == 
    152  * Includes cdscan from the comand-line or from within python 
    153    using the cdat_lite.scripts.cdscan module. 
    154  * Includes the latest version of the British Atmospheric Data 
    155    Centre's cdunifpp module which provides read access to UK 
    156    Met. Office PP files. 
    157  * Downloads and installs Numeric on demand.  If you have Numeric installed 
    158    already but the package doesn't have an egg-info directory easy-install will 
    159    re-install Numeric for you unless you use the --no-deps switch. 
    160  * Currently focused on providing NetCDF and PP format support.  DRS, HDF and DODS (OpeNDAP) 
    161    are disabled in the default egg build. 
    163 == Package Overview == 
    165  * CDAT core packages:  
    166    * cdms, cdtime, cdutil, genutil, Properties, regrid, unidata, xmgrace 
    167    These top-level packages can be used in the same way you would with a full CDAT installation.  E.g. 
    168    {{{ 
    169    #!python 
    170    >>> import cdms 
    171    >>> f ='') 
    172    }}} 
    173  * cdat_lite support packages: These packages support the cdat_lite 
    174    egg installation. 
    175    * cdat_lite.clib: libraries and headers used to build CDAT source: 
    176      libcdms and libnetcdf.   
    177    * cdat_lite.scripts: contains a version of cdscan suitable for declaring as 
    178      a setuptools entry-point.  During installation 
    179      easy-install will create a stub script to execute cdscan on the 
    180      command line 
    181    * cdat_lite.test: contains a test suite. 
    183 == Installation == 
    185 Cdat-lite uses [ easy_install] as its installation mechanism.  This makes downloading and installing the software a one-step process, once you've installed the [ setuptools] package (itself very easy to install).   
    187 Binary eggs and the source tarball of {{{cdat_lite}}} are available from [].  It is also registered at the [ Python Cheeseshop] so should be discovered automatically by {{{easy_install}}}. 
    189  1. Start with a python 2.4+ installation with distutils.  You cannot build or run the egg without distutils.  If your python has been installed by a unix package manager then you may have to install the python-devel package.  To check you've got distutils type the following at the Python prompt: 
    190  {{{ 
    191  >>> import distutils 
    192  >>> 
    193  }}} 
    194  1. Install [ setuptools].  This includes the {{{easy_install}}} script.  The setuptools website has detailed installation instructions but if you have write-permission to your python installation the following should work: 
    195  {{{ 
    196  $ wget 
    197  $ python 
    198  }}} 
    199  1. Now you're ready to install cdat_lite.  Simply type: 
    200  {{{ 
    201  $ easy_install cdat_lite 
    202  }}} 
    203  1. Alternatively, if you want to install into a local directory {{{$EGG_DIR}}}, ensure {{{$EGG_DIR}}} exists and is on your {{{$PYTHONPATH}}} then type: 
    204  {{{ 
    205  $ easy_install -d $EGG_DIR cdat_lite 
    206  }}} 
    207  1. For more installation options see the [ easy_install documentation] 
    210 == Versioning == 
    212 cdat-lite versioning is complicated as it is a repackage of CDAT with 
    213 an updated cdunifpp module. There are therefore 3 version numbers to 
    214 consider: CDAT, cdunifpp and cdat-lite.  Once we start using versions 
    215 of CDAT from the PCMDI SVN the version string could become very long 
    216 indeed therefore the following strategy is used: 
    218  1. Eggs are quoted with the version: <cdat-release>-<cdat-lite-version> 
    219  1. If a PCMDI SVN version of CDAT is used it is stated in long_description not in <cdat-release>. 
    220  1. The cdunifpp version is stated in long_description not in the version.  Any change to the cdunifpp version naturally triggers a new <cdat-lite-version>. 
    224 == Using CDMS scripts == 
    226 cdat_lite includes the libcdms script cdscan within the egg.  You have 
    227 several options to get at it. 
    229 === The stub script === 
    231 When you install the egg with easy_install a script stub is created 
    232 automatically which calls cdscan within the egg.  Therefore you should 
    233 be able to use cdscan as usual.  If you've installed the egg in a 
    234 local directory the script stub will be in that directory. 
    236 === Import cdscan as a module from within python === 
    237 {{{ 
    238 #!python  
    239 >>> import cdat_scripts.cdscan as cdscan 
    240 >>> cdscan.main(['cdscan', '-x', 'dataset.xml', filename, ...]) 
    241 }}} 
    243 The argument to main() must be the equivilent of sys.argv when run 
    244 from the command line, i.e. the first element should be a pseudo 
    245 command name. 
    247 === Import a cdscan entry point using the pkg_resources API. === 
    249 This is the way scripts are supposed to be found from within eggs and is 
    250 used by easy_install to automatically create script stubs. 
    251 {{{ 
    252 #!python 
    253 >>> import pkg_resources 
    254 >>> cdscan = pkg_resources.load_entry_point('cdat_lite', 'console_scripts', 'cdscan') 
    255 >>> cdscan() 
    256 }}} 
    257 In this case sys.argv must be set to the arguments you want.