source: CCCC/trunk/README.txt @ 141

Subversion URL: http://proj.badc.rl.ac.uk/svn/exarch/CCCC/trunk/README.txt@141
Revision 141, 5.3 KB checked in by mjuckes, 7 years ago (diff)

Updated virtual test and correction script generation

Line 
1USAGE
2-----
3
4From the command line:
5----------------------
6
7Required arguments:
8
9python c4.py -p <project> -D <directory>  ## check all files in directory tree, for project in SPECS, CORDEX, CCMI.
10python c4.py -p <project> -d <directory>  ## check all files in directory
11python c4.py -p <project> -f <file>       ## check a single file.
12
13Optional arguments:
14
15  --ld <log file directory>  ## directory to take log files;
16  -R <record file name> ## file name for file to take one record per file checked;
17  --cae                 ## "catch all errors": will trap exceptions and record
18                             in  log files, and then continue. Default is to
19                            stop after unrecognised exceptions.
20  --log <single|multi>  ## Set log file management option -- see "Single log" and "Multi-log" below.
21  --blfmode <mode>      # set mode for batch log file -- see log file modes
22  --flfmode <mode>      # set mode for file-level log file -- see log file modes
23  --aMap                # Read in some attribute mappings and run tests with virtual substitutions, see also map2nco.py
24
25
26Called from python:
27------------------
28The code can also be called from a python script:
29
30import c4
31m = c4.main( args=argList )     # argList is a python list of command line arguments
32if not m.ok:
33  print 'check failed'
34else:
35  print 'success'
36  print 'DRS dictionary:', m.cc.drs    # print drs of last file checked -- not useful in multiple file mode.
37
38e.g.
39m = c4.main( args=[ '-p', 'CORDEX', '-f', dataFilePath, '--ld', logFileDirectory] )
40## run checks on a single file located at dataFilePath, and write logs to logFileDirectory
41
42DEPENDENCIES
43------------
44
45The library uses the cmds2 module to read NetCDF files.
46
47OUTPUT
48------
49
50Single log (default for single file): 
51  -- log of errors found and checks passed
52  -- "Rec.txt" -- single record summarising results. If no errors are found, the archive directory path for the file will be in this record.
53
54Multi-log (default for multiple files):
55  -- separate log of errors for each file;
56  -- summary log, 3 records per file;
57  -- "Rec.txt" -- single record for each file, as above
58
59Log file modes.
60Valid modes are: 'a': append
61                 'n', 'np': new file, 'np': protect after closing (mode = 444)
62                 'w', 'wo': write (overwrite if present), 'wo': protect after closing (mode = 444)
63
64Note that the log files generated in multi-log mode will re-use file names. If running with --flfmode set to 'n','np' or 'wo' it will be necessary to change or clear the target directory. The names of batch log files include the time, to the nearest second, when the process is started, so will not generally suffer from re-use.
65
66Vocabulary lists GCMModelName.txt and RCMModelName.txt are held on the DMI CORDEX site:
67
68  http://cordex.dmi.dk/joomla/images/CORDEX/GCMModelName.txt
69  http://cordex.dmi.dk/joomla/images/CORDEX/RCMModelName.txt
70
71To update the CMOR tables use:
72"git clone git://uv-cdat.llnl.gov/gitweb/cordex-cmor-tables.git"
73
74VIRTUAL MODE
75------------
76
77The virtual mode can be used to validate substituions before adjusting systems which have been used to generate data, or as the first step of a procedure for repairing some classes of errors.
78
79To use this mode, a mapping file is needed. This can be generated by an initial run of the checker with no virtual substitutions. A file naemd "amapDraft.txt" will be generated. This file should be inspected to ensure that suggested changes make sense.
80
81A typical directive will be of the form:
82@var=rlus;standard_name=surface_upward_longwave_flux_in_air|standard_name=surface_upwelling_longwave_flux_in_air
83
84The meaning is: for variable "rlus", set the attribute "standard_name" to "surface_upwelling_longwave_flux_in_air" where the input file has "surface_upward_longwave_flux_in_air".
85
86If run with the --aMap flag, the checker will test attributes after making virtual substituions. I.e. there are no changes made to the files at this stage, but results of the tests apply as if changes have been made.
87
88After running in virtual mode, c4.py will generate a file named "attributeMappingsLog.txt" which contains a record for every change to every file. If the results of running in virtual mode are positive, this file can be used to create a script to modify the files, by running "amap2nco.py":
89
90python amap2nco.py attributeMappingsLog.txt /tmp/batch1 /tmp/batch1_corrected
91## this will generate a list of NCO commands in "ncoscript.sh", which will apply the changes and create new files in "/tmp/batch1_corrected".
92
93It is recommended that the data values in the corrected files should be checked after running this script.
94
95By default, the amap2nco.py program will generate commands to modify the tracking_id and creation_date global attributes at the same time as making other changed. The "history" attribute is modified by the NCO library.
96
97EXCEPTIONS
98----------
99The exception handling is designed to ensure that problems analysing one file do not prevent testing of other files.
100Traceback information is written to log file.
101
102BUGS
103----
104The cmds2 library generates a data array for dimensions if there is none present in the file. Tests applied to this library generated array will generate mis-leading error messages. Within cmds2 there is no clear method of distinguishing between library generates arrays and those which exist in the data file. The solution may be to move to using the NetCDF4 module instead.
105
106----------
Note: See TracBrowser for help on using the repository browser.