wiki:PackageFCC

Version 1 (modified by mjuckes, 6 years ago) (diff)

--

File Compliance Checker

Climate model datasets are usually composed of large numbers of data files. Efficient use of these files is promoted by use of standard formats and project specific profiles of these formats. For the climate model intercomparisons carried out under the auspice of the World Climate Research Programme standardisation has been promoted through use of a community software package, CMOR, developed and maintained by PCMDI. This package generates NetCDF files with standardised use of dimensions and specified attributes. Despite adopting use of CMOR, problems do occur in large data collections: -- the scientist using CMOR may not be familiar with all options and may specify incorrect parameters; -- files may be modified using scripting tools after generation by CMOR (e.g. to correct minor errors in the header without re-running the time consuming CMOR process); -- merging of different processing batches to produce a collection of files for publication can introduce inconsistencies.

In other intercomparison projects, CORDEX for instance, use of software to standardise data files is not mandated.

The FCC package is designed to provide a simple check of file compliance with a range of specifications. It has initially been developed to test compliance of CORDEX data with the requirements ([  http://cordex.dmi.dk/joomla/images/CORDEX/cordex_archive_specifications.pdf]), but with flexibility to be adapted for other requirements specifications.

Download and usage

The script is designed to be easy to use with minimal installation requirements. It is currently held in a "subversion" repository ( http://subversion.apache.org ).

{{{svn co  http://proj.badc.rl.ac.uk/svn/exarch/FCC2/trunk/}}

will download the code into a local directory "trunk". Further details are in trunk/README. To test the installation go to trunk/work and execute:

./fcc -d examples_bad/ -c config_cordex.txt --nlp --oic --R -l examples_bad_test.log

The message "SCRIPT COMPLETED:: TEST(S) FAILED" should be displayed on the command line.

Further developments

The syntax of the configuration file needs to be improved to add flexibility for use with other data requirements.

Attachments