wiki:PackageFCC

The CEDA Compliance Checker

30th June 2014: Version 1.0 released. This version provides improved installation procedures and follows the implementation of a better release management process.

Available from  http://proj.badc.rl.ac.uk/svn/exarch/CCCC/trunk/ or  https://pypi.python.org/pypi/ceda-cc

The CEDA Compliance Checker (CEDA CC) is a re-configuration of FCC2 (see below), with emphasis on clean programme structure and error handling, at the expense of flexibility of configuration. CEDA CC provides a more complete check of the CORDEX requirements [see http://proj.badc.rl.ac.uk/exarch/attachment/wiki/PackageFCC/ESGF_CORDEX_data_checks.pdf].

The package is currently configured to check CMIP5, CORDEX, SPECS and CCMI data.

Usage:

  • ceda-cc -p <project> -D <directory> # check all data in directory tree under specified directory
  • ceda-cc -p <project> -d <directory> # check all data in specified directory
  • ceda-cc -p <project> -f <file> # check a single file

If running directly from the installation directory, "ceda_cc" in the above examples should be replaced by "python ceda_cc/c4.py". This replaces the formulation "python c4.py ...." used prior to version 1.0. All other argument options are unchanged.

See the "README" file in the repository for more details -- there is also a python library API.

File Compliance Checker

The FCC script is an earlier version of the Ceda Compliance Checker, and is no longer supported.

Climate model datasets are usually composed of large numbers of data files. Efficient use of these files is promoted by use of standard formats and project specific profiles of these formats. For the climate model intercomparisons carried out under the auspice of the World Climate Research Programme standardisation has been promoted through use of a community software package, CMOR, developed and maintained by PCMDI. This package generates NetCDF files with standardised use of dimensions and specified attributes. Despite adopting use of CMOR, problems do occur in large data collections: -- the scientist using CMOR may not be familiar with all options and may specify incorrect parameters; -- files may be modified using scripting tools after generation by CMOR (e.g. to correct minor errors in the header without re-running the time consuming CMOR process); -- merging of different processing batches to produce a collection of files for publication can introduce inconsistencies.

In other intercomparison projects, CORDEX for instance, use of software to standardise data files is not mandated.

The FCC package is designed to provide a simple check of file compliance with a range of specifications. It has initially been developed to test compliance of CORDEX data with the requirements (  http://cordex.dmi.dk/joomla/images/CORDEX/cordex_archive_specifications.pdf), but with flexibility to be adapted for other requirements specifications.

Download and usage

The script is designed to be easy to use with minimal installation requirements. It is currently held in a "subversion" repository ( http://subversion.apache.org ).

svn co http://proj.badc.rl.ac.uk/svn/exarch/FCC2/trunk/

will download the code into a local directory "trunk". Further details are in trunk/README. To test the installation go to trunk/work and execute:

./fcc -d examples_bad/ -c config_cordex.txt --nlp --oic --R -l examples_bad_test.log

The message "SCRIPT COMPLETED:: TEST(S) FAILED" should be displayed on the command line.

Contact

For queries about the compliance checker, contact martin.juckes at stfc.ac.uk.

Attachments