wiki:CdatLite

Version 16 (modified by domlowe, 12 years ago) (diff)

Added note on 64 bit install problem

NEW! CDAT_LITE 5.0

cdat_lite-5.0 is now available. Below are some notes on what the new version brings or go to the 4.3 documentation

cdat_lite-5.0

Version 5.0 of cdat_lite introduces many changes.

  1. It is built directly from the CDAT-5.0 codebase and version numbers are completely synchronised. CDAT-5.0 requires Python-2.5 and uses numpy rather than Numeric. See Upgrading for what you need to do to your code.
  2. It is now only distributed as a tarball. Binary distributions tended to run into library version conflicts on different versions of linux. However easy_install should still be able to compile the package automatically. See Installing. Also you can make your own binary eggs for given OSes.
  3. You now need to have the netcdf library pre-installed. Automatically building netcdf wasn't portable enough.

Installing

First satisfy a couple of dependencies.

Make sure you have the  netcdf libraries installed. If they are in a non-standard location make a note of it (you will be prompted) or set NETCDF_DIR to its installation prefix.

Install numpy. If you aren't bothered about linear algebra optimisation this usually works with easy_install. E.g. 

$ easy_install numpy

Now you are ready to install cdat_lite. On many linux systems the C compiler is incorrectly detected therefore it is safest to do: 

$ CC=gcc easy_install cdat_lite

Advanced Installation

You might want more control over the installation process but you can still get easy_install to download the tarball for you: 

$ easy_install -eb <dir> cdat_lite

This will download and unpack the tarball into <dir> then you can run the setup.py script directly or build a binary egg for installation elsewhere.

Upgrading

The move from Python-2.4 to Python-2.5 and from Numeric to numpy requires very few changes to your code. Most changes that are needed can be done automatically using the convertcdms script. 

$ convertcdms
Usage:
    convertcdms.py [options] path

      Convert scripts that use the Numeric, MA, cdms, and regrid modules to numpy, ma, cdms2, and regrid2.

    -or-

    convertcdms.py [options] -r direc

      Convert all Numeric/MA/cdms Python scripts and C source code in directory 'direc' to numpy/ma/cdms2.

      Use this form to convert C source code.

Arguments:

    path: A .py file to convert
    direc: A directory: convert all .py, .c, and .h files

Options:

    -a:           Disable 'aggressive' conversion (see Note 1)
    -c:           Clobber the original file(s). The default is to move the original to <name>.orig
    -h:           Print a help message
    -n:           Do not write '# Adapted by...' comment at the top of file.
    -r:           Recursively convert all .py and .c/.h files in directory direc
    -s direc:     Skip the directory. This option may be used more than once.
                    For example, '-s .svn' skips all subversion subdirectories.

Notes:

    1) The -a option turns off the following translations:

        - MA.Float => Numeric.Float, similarly for MA.Int, MA.NewAxis
        - MA.Numeric => Numeric
        - The 'typecode=' argument in MA functions is changed to 'dtype='
        - XX.mask() => XX.mask
        - XX.mask is None => ((XX.mask is None) or (XX.mask is MV2.nomask))
        - A keyword argument 'axis=0' is added to MA.sum, MA.average, MA.product, and MA.repeat

    2) By default the following translations are made:

        - import cdms => import cdms2 as cdms
        - import regrid => import regrid2 as regrid
        - import MV => import MV2 as MV
        - import Numeric => import numpy.oldnumeric as Numeric
        - import MA => import numpy.oldnumeric.ma as MA
        - import cdms.MV => import cdms2.MV2
        - from cdms import XX => from cdms2 import XX (similarly for regrid, MV, MA, and Numeric)
        - from cdms.XX import YY => from cdms2.XX import YY (similarly for regrid, MV, MA, and Numeric)
        - import cdms as XX => import cdms2 as XX (similarly for regrid, MV, MA, and Numeric)
        - import cdms.XX => import cdms2.XX (similarly for regrid, MV, MA, and Numeric)
        - import XX, cdms, YY => import XX, cdms2 as cdms, YY (similarly for regrid, MV, MA, and Numeric)
        - the translations in Note (1)
        - the translations in numpy.oldnumeric.alter_code1. This module is used for most Numeric and MA-related translations.

A few minor changes may also be required. If you use arrays in logical expressions you may need to change your code. The following would work with Numeric: 

>>> import numpy
>>> x = numpy.arange(4)
>>> if x:
...   print 'hello'
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
>>>

Installation problems (and solutions)

On a 64-bit machine you may get a message like: 

`a local symbol' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/3.4.6/../../../../lib/libnetcdf.a: could not read symbols: Bad value

In which case you need to recompile your NetCDF installation to use the fPIC option as follows: 

./configure --with-pic
make
make install

CDAT-lite 4.3

cdat-lite is a simple repackaging of the I/O layer of the  Climate Data Analysis Tools (CDAT) as a Python Egg?. A short article on the reasons behind cdat-lite and some of the design decisions made can be found  here on  Stephen Pascoe's blog.

Features

  • Includes cdscan from the comand-line or from within python using the cdat_lite.scripts.cdscan module.
  • Includes the latest version of the British Atmospheric Data Centre's cdunifpp module which provides read access to UK Met. Office PP files.
  • Downloads and installs Numeric on demand. If you have Numeric installed already but the package doesn't have an egg-info directory easy-install will re-install Numeric for you unless you use the --no-deps switch.
  • Currently focused on providing NetCDF and PP format support. DRS, HDF and DODS (OpeNDAP) are disabled in the default egg build.

Package Overview

  • CDAT core packages:
    • cdms, cdtime, cdutil, genutil, Properties, regrid, unidata, xmgrace These top-level packages can be used in the same way you would with a full CDAT installation. E.g. 
      >>> import cdms
>>> f = cdms.open('dataset.nc')
  • cdat_lite support packages: These packages support the cdat_lite egg installation.
    • cdat_lite.clib: libraries and headers used to build CDAT source: libcdms and libnetcdf.
    • cdat_lite.scripts: contains a version of cdscan suitable for declaring as a setuptools entry-point. During installation easy-install will create a stub script to execute cdscan on the command line
    • cdat_lite.test: contains a test suite.

Installation

Cdat-lite uses  easy_install as its installation mechanism. This makes downloading and installing the software a one-step process, once you've installed the  setuptools package (itself very easy to install).

Binary eggs and the source tarball of cdat_lite are available from http://ndg.nerc.ac.uk/dist. It is also registered at the  Python Cheeseshop so should be discovered automatically by easy_install.

  1. Start with a python 2.4+ installation with distutils. You cannot build or run the egg without distutils. If your python has been installed by a unix package manager then you may have to install the python-devel package. To check you've got distutils type the following at the Python prompt: 
    >>> import distutils
>>>
  2. Install  setuptools. This includes the easy_install script. The setuptools website has detailed installation instructions but if you have write-permission to your python installation the following should work: 
    $ wget http://peak.telecommunity.com/dist/ez_setup.py
$ python ez_setup.py
  3. Now you're ready to install cdat_lite. Simply type: 
    $ easy_install cdat_lite
  4. Alternatively, if you want to install into a local directory $EGG_DIR, ensure $EGG_DIR exists and is on your $PYTHONPATH then type: 
    $ easy_install -d $EGG_DIR cdat_lite
  5. For more installation options see the  easy_install documentation

Versioning

cdat-lite versioning is complicated as it is a repackage of CDAT with an updated cdunifpp module. There are therefore 3 version numbers to consider: CDAT, cdunifpp and cdat-lite. Once we start using versions of CDAT from the PCMDI SVN the version string could become very long indeed therefore the following strategy is used:

  1. Eggs are quoted with the version: <cdat-release>-<cdat-lite-version>
  2. If a PCMDI SVN version of CDAT is used it is stated in long_description not in <cdat-release>.
  3. The cdunifpp version is stated in long_description not in the version. Any change to the cdunifpp version naturally triggers a new <cdat-lite-version>.

Using CDMS scripts

cdat_lite includes the libcdms script cdscan within the egg. You have several options to get at it.

The stub script

When you install the egg with easy_install a script stub is created automatically which calls cdscan within the egg. Therefore you should be able to use cdscan as usual. If you've installed the egg in a local directory the script stub will be in that directory.

Import cdscan as a module from within python

>>> import cdat_scripts.cdscan as cdscan
>>> cdscan.main(['cdscan', '-x', 'dataset.xml', filename, ...])

The argument to main() must be the equivilent of sys.argv when run from the command line, i.e. the first element should be a pseudo command name.

Import a cdscan entry point using the pkg_resources API.

This is the way scripts are supposed to be found from within eggs and is used by easy_install to automatically create script stubs. 

>>> import pkg_resources
>>> cdscan = pkg_resources.load_entry_point('cdat_lite', 'console_scripts', 'cdscan')
>>> cdscan()

In this case sys.argv must be set to the arguments you want.