source: CCCC/trunk/README.txt @ 180

Subversion URL: http://proj.badc.rl.ac.uk/svn/exarch/CCCC/trunk/README.txt@180
Revision 180, 8.0 KB checked in by mjuckes, 6 years ago (diff)

merged from branches-mnj (which was branched from branches-sp)

Line 
1USAGE
2-----
3
4From the command line:
5----------------------
6
7Required arguments:
8
9python ceda_cc/c4.py -p <project> -D <directory>  ## check all files in directory tree, for project in SPECS, CORDEX, CCMI, CMIP5.
10python ceda_cc/c4.py -p <project> -d <directory>  ## check all files in directory
11python ceda_cc/c4.py -p <project> -f <file>       ## check a single file.
12python ceda_cc/c4.py --copy-config <dest-dir>     ## copy the default configuration directory to <dest-dir> to enable customisation.
13
14Optional arguments:
15
16  --ld <log file directory>  ## directory to take log files;
17  -R <record file name> ## file name for file to take one record per file checked;
18  --cae                 ## "catch all errors": will trap exceptions and record
19                             in  log files, and then continue. Default is to
20                            stop after unrecognised exceptions.
21  --log <single|multi>  ## Set log file management option -- see "Single log" and "Multi-log" below.
22  --blfmode <mode>      # set mode for batch log file -- see log file modes
23  --flfmode <mode>      # set mode for file-level log file -- see log file modes
24  --aMap                # Read in some attribute mappings and run tests with virtual substitutions, see also map2nco.py
25
26Environment variables:
27
28  CC_CONFIG_DIR  ## Set to the location of a custom configuration directory.  If unset the default configuration will be used.
29
30After running:
31
32The log file directory may contain hundreds of files with reports of errors. To get a summary, run:
33
34python summary.py <log file directory>
35
36This will produce a listing of errors, the number of times they occur and up to two of the files which contain the error. It is hoped that inspection of one or 2 files will provide enough information to trace the problems which lead to the error reports.
37
38
39Installing as a package:
40------------------------
41
42You can also install the code into your Python environment and then use the "ceda-cc" command to invoke c4.py with the same arguments ans described above.
43
44 1. If you have "pip" installed simply execute:
45    $ pip install ceda-cc
46    or after downloading the tarball
47    $ pip install ceda-cc-$VERSION.tar.gz
48
49 2. If you have the setuptools package you can execute the following from the distribution directory:
50    $ python setup.py install
51
52If you install ceda-cc in this way you can use the --copy-config command to export the default configuration into a directory where you can edit the configuration.
53
54
55Called from python:
56------------------
57The code can also be called from a python script:
58
59from ceda_cc import c4
60m = c4.main( args=argList )     # argList is a python list of command line arguments
61if not m.ok:
62  print 'check failed'
63else:
64  print 'success'
65  print 'DRS dictionary:', m.cc.drs    # print drs of last file checked -- not useful in multiple file mode.
66
67e.g.
68m = c4.main( args=[ '-p', 'CORDEX', '-f', dataFilePath, '--ld', logFileDirectory] )
69## run checks on a single file located at dataFilePath, and write logs to logFileDirectory
70
71DEPENDENCIES
72------------
73
74The library can uses the cdms2, python-netCDF4 or Scientific module to read NetCDF files.
75By default, it will use the cdms2 module if available. Support for the netCDF4 and Scientific modules has been added recently.
76To change the default, change the order in which modules are listed in the "supportedNetcdf" list in file_utils.py
77
78Is available as part of the cdat-lite package (http://proj.badc.rl.ac.uk/cedaservices/wiki/CdatLite ).
79For python-netCDF4, see http://code.google.com/p/netcdf4-python/.
80For Scientific see http://dirac.cnrs-orleans.fr/plone/software/scientificpython/  . Note that search engines confuse "ScientificPython" with "SciPy". The SciPy package also contains a netcdf API, but when tested in April 2014 this could not read data from NetCDF 4 files, and so is not supported here.
81
82OUTPUT
83------
84
85Single log (default for single file): 
86  -- log of errors found and checks passed
87  -- "Rec.txt" -- single record summarising results. If no errors are found, the archive directory path for the file will be in this record.
88
89Multi-log (default for multiple files):
90  -- separate log of errors for each file;
91  -- summary log, 3 records per file;
92  -- "Rec.txt" -- single record for each file, as above
93
94Log file modes.
95Valid modes are: 'a': append
96                 'n', 'np': new file, 'np': protect after closing (mode = 444)
97                 'w', 'wo': write (overwrite if present), 'wo': protect after closing (mode = 444)
98
99Note that the log files generated in multi-log mode will re-use file names. If running with --flfmode set to 'n','np' or 'wo' it will be necessary to change or clear the target directory. The names of batch log files include the time, to the nearest second, when the process is started, so will not generally suffer from re-use.
100
101Vocabulary lists GCMModelName.txt and RCMModelName.txt are held on the DMI CORDEX site:
102
103  http://cordex.dmi.dk/joomla/images/CORDEX/GCMModelName.txt
104  http://cordex.dmi.dk/joomla/images/CORDEX/RCMModelName.txt
105
106To update the CMOR tables use:
107"git clone https://github.com/PCMDI/cordex-cmor-tables"
108
109VIRTUAL MODE
110------------
111
112The virtual mode can be used to validate substituions before adjusting systems which have been used to generate data, or as the first step of a procedure for repairing some classes of errors.
113
114To use this mode, a mapping file is needed. This can be generated by an initial run of the checker with no virtual substitutions. A file named "amapDraft.txt" will be generated. This file should be inspected to ensure that suggested changes make sense.
115
116A typical directive will be of the form:
117@var=rlus;standard_name=surface_upward_longwave_flux_in_air|standard_name=surface_upwelling_longwave_flux_in_air
118
119The meaning is: for variable "rlus", set the attribute "standard_name" to "surface_upwelling_longwave_flux_in_air" where the input file has "surface_upward_longwave_flux_in_air".
120
121"amapDraft.txt" should be copied to a new location before running in virtual mode. This draft will only contain directives for errors if the corect value is unique. The suggested corrections to variable attributes will make these consistent with the variable name. If the inconsistency has arisen because a variable has been given the wrong name this will exaggerate the problem rather than solving it. All changes should be checked.
122
123Additional directives can be added. e.g.
124@;institute_id=mohc|institute_id=MOHC
125will instruct the code to replace "mohc" with "MOHC" in the global attribute "institute_id".
126
127If run with the --aMap flag, the checker will test attributes after making virtual substituions. I.e. there are no changes made to the files at this stage, but results of the tests apply as if changes have been made.
128
129After running in virtual mode, c4.py will generate a file named "attributeMappingsLog.txt" which contains a record for every change to every file. If the results of running in virtual mode are positive, this file can be used to create a script to modify the files, by running "amap2nco.py":
130
131python amap2nco.py attributeMappingsLog.txt /tmp/batch1 /tmp/batch1_corrected
132## this will generate a list of NCO commands in "ncoscript.sh", which will apply the changes and create new files in "/tmp/batch1_corrected".
133
134It is recommended that the data values in the corrected files should be checked after running this script.
135
136By default, the amap2nco.py program will generate commands to modify the tracking_id and creation_date global attributes at the same time as making other changed. The "history" attribute is modified by the NCO library.
137
138EXCEPTIONS
139----------
140The exception handling is designed to ensure that problems analysing one file do not prevent testing of other files.
141Traceback information is written to log file.
142
143BUGS
144----
145The cmds2 library generates a data array for dimensions if there is none present in the file. Tests applied to this library generated array will generate mis-leading error messages. Within cmds2 there is no clear method of distinguishing between library generates arrays and those which exist in the data file. The solution may be to move to using the NetCDF4 module instead.
146
147----------
Note: See TracBrowser for help on using the repository browser.