wiki:CdatLite

Version 28 (modified by spascoe, 9 years ago) (diff)

--

NEWS

2010-03-16
cdat_lite-5.2-1.tar.gz is available. This release includes minor fixes to enable cdat_lite to build with  pip.
2010-01-12
There is a  significant bug in numpy-1.4.0 affecting the MaskedArray implementation. Therefore cdat_lite shouldn't be used with numpy-1.4.0.
2009-09-23
cdat-lite-5.2 has been released. Select you favourite method of installation:
  1. Use easy_install cdat-lite
  2. Download Manually from the NDG Egg repository
  3. Visit the  cdat-lite page on the Python Package Index
2009-09-15
cdat-lite-5.2rc1 is out and  available on pypi or the NDG Egg repository

CDAT-Lite

CDAT-Lite is a Python package for managing and analysing climate science data. It is a subset of the Climate Data Analysis Tools (CDAT) developed by PCMDI at Lawrence Livermore National Laboratory.

CDAT-lite aims to compliment CDAT by focussing on it's core data management and analysis components and by offering a radically different installation system to CDAT. As a result it is much more lightweight (hence the name): CDAT's source distribution is the order of 1Gb whereas CDAT-lite is under 5Mb.

1   Installing cdat-lite

cdat-lite is distributed as a tarball available from the cdat-lite homepage on the NERC Data Grid wiki . It is also installable using the easy_install tool. If you are familiar with easy_install try this super-quick installation recipe:

$ export NETCDF_HOME=/usr/local/netcdf
# Required if using a NetCDF4 compiled with HDF5
$ export HDF5_HOME=/usr/local/hdf5
$ easy_install cdat_lite

1.1   Dependencies

To install cdat-lite you will need:

1. Python 2.5.x. cdat-lite has not been tested on 2.6 but may work (feedback would be gratefully received). It is unlikely to work on 3.0.

2. setuptools. cdat-lite will attempt to download and install setuptools if it is missing but it is safer to install it first.

3. NetCDF-3.x or greater. cdat-lite should work with any relatively modern NetCDF3 installation on your system provided it is compiled as a shared library. It will also work with NetCDF4 configured in various different ways, including embedded OPeNDAP mode.

  1. If you want to run the short test suite you will need nose

1.2   Selecting your NetCDF installation

cdat-lite will work with NetCDF3 or NetCDF4 but because it is referenced by shared libraries (the python C extension modules) it must be compiled as position independent code. It is probably easiest to install NetCDF as a shared library (use --enable-shared in the NetCDF configure script). Alternatively, you can configure NetCDF with:

$ ./configure --with-pic ...

If you are using NetCDF4 you will also need to configure HDF5 with --enable-shared or --with-pic.

cdat-lite will look for a NetCDF installation in several places and prompt you if it can't find it. If your NetCDF is installed somewhere unusual, or if you want to select a specific installation, set the NETCDF_HOME variable. E.g.:

# sh users
$ export NETCDF_HOME=/usr/local/netcdf
# csh users
$ setenv NETCDF_HOME /usr/local/netcdf

If you are using NetCDF4 cdat-lite will also look for your HDF5 installation which you can configure in a similar way:

# sh users
$ export HDF5_HOME=/usr/local/hdf5
# csh users
$ setenv HDF5_HOME /usr/local/hdf5

Note, you don't need these environment variables set to run cdat_lite, although the libraries must be findable by your system's dynamic linker. This can be configured by setting LD_LIBRARY_PATH or using ldconfig.

1.3   Running the installer

If you have all the dependencies in place you can try using easy_install to automatically download and install cdat_lite. Make sure you have access to the internet, with the appropriate HTTP proxy settings, and do:

$ easy_install cdat-lite

Alternatively you might want to see what you are installing :-). In this case either download the tarball or use easy_install to do it for you:

$ easy_install -eb . cdat-lite
# The cdat-lite tarball will be downloaded unpacked into you current directory

Now from the distribution directory run the build and install steps separately:

$ python setup.py bdist_egg
$ easy_install dist/cdat-lite*.egg

1.4   Installing as an unprivileged user

If you don't have write access to your python distribution you can use the tool virtualenv to create a local python environment with it's own easy_install executable which you can then use to install cdat-lite. In combination with NETCDF_HOME, HDF5_HOME and LD_LIBRARY_PATH it should be possible to install all dependencies of cdat-lite locally. See the virtualenv for details on installation or try this recipe after downloading the virtualenv:

# From virtualenv distribution directory
$ ./virtualenv.py <virtualenv-path>
$ cd <virtualenv-path>
$ source bin/activate
(venv)$ easy_install cdat-lite

2   Testing the installation

cdat-lite ships with a small set of tests designed to verify that it has been built successfuly. These tests require the testing framework nose. Once cdat-lite is installed just run:

$ nosetests cdat_lite

When run from cdat-lite's distribution directory nosetests will run slightly differently, running some tests that are known to fail at the moment. To disable this behaviour do:

$ nosetests --config=''

3   FAQ

3.1   What is CDAT?

CDAT is a large suite of open source tools distributed by PCMDI for the management and analysis of climate data. It includes several visualisation components and the graphical user interface VCDAT.

3.2   What is the difference between CDAT and cdat-lite?

Differences between CDAT and CDAT-lite can be classified as differences in scope, i.e. which packages are included, and installation system.

cdat-lite contains the 'cdms2' package and a few related packages. It does not include the 'vcs' visualisation package or the VCDAT graphical user interface. As of v5.1.1-0.3pre3 the included packages are:

  • cdms2
  • cdtime
  • cdutil
  • genutil
  • ncml
  • Properties
  • regrid2
  • unidataa
  • xmgrace

CDAT bundles virtually all dependencies together in it's source distribution -- even Python itself. This has it's advantages as it simplifies satisfying dependencies and avoids version conflicts between dependencies. However, if you want to integrate CDAT's data management components into your existing Python architecture CDAT can be overkill.

3.3   What has changed between cdat-lite-4.x and cdat-lite-5.x?

If you are a cdat-lite-4 user (or a CDAT 4 user) you have a big migration job on your hands. CDAT-4 uses the Numeric package for arrays which has been out of date and unmaintained for a long time now. It is known to have problems on 64bit architectures.

3.4   How does cdat-lite track changes to CDAT?

cdat-lite tries to release major new versions shortly after new versions of CDAT. Sometimes CDAT-trunk contains important fixes that should be applied so that the latest cdat_lite can run ahead of official CDAT releases (although sometimes CDAT recommends you build from trunk anyway).

The one exception is the UK Met. Office PP file support which is usually updated in cdat_lite before CDAT. In all cases the exact build versions of CDAT and cdunifpp will be stated in the distribution's setup.py file.

3.5   How can I use CMOR2 with cdat-lite?

We are interested to hear any with experience of using CMOR2 with cdat-lite but it should be as simple as downloading the distribution and installing it in parallel with:

# From the CMOR install directory
$ python setup.py install

3.6   How can I use OPeNDAP with cdat-lite?

OPeNDAP support is an experimental feature of cdat-lite at the moment. Unlike CDAT you don't select OPeNDAP explicitly during installation but cdat-lite will inherit any OPeNDAP support embedded into the NetCDF4 library. Recent beta releases of NetCDF4 provides a switch to transparently use OPeNDAP.