source: nappy/trunk/nappy/2008 refactoring notes.txt @ 3359

Subversion URL: http://proj.badc.rl.ac.uk/svn/ndg/nappy/trunk/nappy/2008 refactoring notes.txt@3359
Revision 3359, 12.9 KB checked in by astephen, 12 years ago (diff)
Line 
1= Changes needed to make Nappy fit for purpose =
2
3 * Need error checking throughout - making it more robust but also checking on read will make Nappy a useful NASA Ames checker replacement.
4
5 * cdms interface should be in a separate part of the stack, not in core classes AND SUB-CLASSES
6
7 * Need to make an egg of it, but in the meantime we also need it installed as:
8
9-------------------
10We need to decide what is required in terms of:
11
12    * actual formats
13    * mappings to/from NetCDF
14    * mapping to/from other formats (if required).
15
16    AS and CK have been analysing the required changes to make Nappy suitable for automated conversion of NetCDF-CF in DCIP to NASA Ames / CSV variant.
17
18        Here is an initial list of issues
19
20    * Put in the option to order variables explicitly, rather than relying on nasa_ames_var_number attributes.
21
22    * Support for 2110 – NX for the quickest changing IV needs to be converted to an auxiliary variable which does not fit nicely into the structure. In principle the second independent variable length changes at each first independent variable value but we may need it fixed.
23
24    * Refactoring to make it clearer and functions shorter but not too short.
25    * Add an option for the learning column.
26    * Proper use of FFI selection.
27    * In cdms2na.py, stopping filling an axis once the start and increment are known may not be enough. May need to populate all of the axis.
28    * Column headings for Excel users, this will clash with the current #End of normal comments# lines.
29    * Is it still right that the 4010 class is the same as 2010?
30    * Does comma separation option work correctly – any consequences for RDATE and DATE which retains some space separation?
31    * float rounding in nappy to get difference between independent variable values - and degree of accuracy (8.3f etc)
32    * cdms2na() needs argument "variable_order" that is a list of the order you want them to appear in 2110.
33    * Need sensible handling of rotated grid data where found (might be wrapper outside of nappy
34    * FFI 2110 is best for 2D columns for Excel etc. - Nappy must support this
35
36--------------------
37
38
39Dear Ag Stephans
40
41I found your nappy library for reading Ames Files at http://home.badc.rl.ac.uk/astephens/software/nappy/
42For one of my projects I want to use that lib in a plugin for a wiki (http://moinmo.in) to give some feedback about submitted/attached data files.
43
44Your web page states last modified at Wed, 06 Apr 2005 13:44:53 GMT
45 As I have seen on http://home.badc.rl.ac.uk/astephens/software/nappy/USAGE.txt
46"[NOTE: We plan to implement a getVariableArray(var_name) to grab a specific variable from the above array.]"
47
48I like to ask if there is probably a further version of that lib available?
49
50cheers
51Reimar Bauer
52
53----------------------
54
55
56=== Adding annotation column ===
57
58via argument: annotated=True|False
59
60Only on output. So only affects write methods.
61NOTE: could do read as well by reading in file and then removing column 1 (as long as we know the delimiter is a comma). But don't do this now.
62
63Ask Charles to come up with a list of definitions for each column in a config file.
64
65Need to defined names in config file and a ways of mapping to each of the lines by tagging to self.A, self.X self.XNAMES etc
66
67=== Adding CSV ===
68
69Via delimiter/spacer argument. Would be nice to have a writeCSV() method.
70
71Should 'csv' and 'delim' args be sent to write methods rather than __init__()?
72---
73
74===
75
76Do we need global DEBUG = True|False
77
78Then all the prints could be:
79
80if DEBUG == True: print "blah"
81
82===============
83localRules.py --> REMOVE COMPLETELY
84localRules/blah - remove this and put stuff in a top-level config file, or even remove altogether.
85
86localRules/aircraft.py - Need to consider how this can be a sub-class of Cdms2NA so that we push all the odd code into separate modules. Need to encapstulate the differences into one or two methods that are small and can be overridden.
87 * decided to dump all aircraft stuff in  an unsupported dir without refactoring in - it will probably never be used!
88
89=================
90
91import nappy  (via nappy_api.py module)
92nappy.convertNAToNC(na_file, nc_file)
93nappy.convertNCToNA(nc_file, na_file)
94
95================
96
97Have I broken textParser.py's main function - is it same in old and new - need a test for it!
98
99=====
100
101_readData[12] are crying out for useful names
102
103Should we leave in the interactive time units checker in na_to_cdms.py - ask Charles
104
105Unit tests
106==========
107
1081001 appears to be done. Others are there as stubs but all need writing.
109
110We then need a set of nc_interface tests as well! Need NetCDF files to convert the other way as well. Need to be small.
111
112----
113
114=== Current state of Unit tests ===
115
116Successful:
117 * test_na_file_1001.py
118 * test_na_file_1010.py
119 * test_na_file_2010.py
120 * test_na_file_2110.py
121 * test_na_file_2310.py
122
123Failed:
124 * test_na_file_1020.py
125 * test_na_file_2160.py
126 * test_na_file_3010.py
127 * test_na_file_4010.py
128
129Did they work before in old nappy.
130
131Old nappy is available at:
132
133http://proj.badc.rl.ac.uk/ndg/browser/nappy/tags/nappy_pre_refactor_feb2008/nappy
134
135Get it with:
136
137svn co svn+ssh://proj.badc.rl.ac.uk/svn/ndg/nappy/tags/nappy_pre_refactor_feb2008/nappy
138
139
140All need to compare exactly to old output from old nappy!
141
142----
143
144Get new version from:
145
146svn co svn+ssh://proj.badc.rl.ac.uk/svn/ndg/nappy/trunk
147
148-------
149
150= Changes needed to make Nappy fit for purpose =
151
152
153 * Need error checking throughout - making it more robust but also checking on read will make Nappy a useful NASA Ames checker replacement.
154
155 * cdms interface should be in a separate part of the stack, not in core classes AND SUB-CLASSES
156
157 * Need to make an egg of it, but in the meantime we also need it installed as:
158
159   * nappy-0.2.3
160     * nappy
161     * bin
162     * nappy/nc_interface
163     * nappy/cdms_utils/
164     * nappy/utils
165     * nappy/na_file
166     * nappy/contrib/aircraft
167   
168
169 * GET REST FROM WIKI!
170
171
172===============
173localRules.py --> REMOVE COMPLETELY
174localRules/blah - remove this and put stuff in a top-level config file, or even remove altogether.
175
176localRules/aircraft.py - Need to consider how this can be a sub-class of Cdms2NA so that we push all the odd code into separate modules. Need to encapstulate the differences into one or two methods that are small and can be overridden.
177
178======
179bin/scanFAAM.py - put in contrib
180
181======
182
183version.py - put in config file.
184
185======
186
187general.py --> call it utils/xxxxx.py
188
189textParser --> utils/text_parser.py
190
191naError.py --> na_error/na_error.py
192
193naCore.py --> na_file/na_core.py
194
195listManipulator -_> utils/list_manipulator.py
196
197cdmsMap.py --> put in config file given simplicity
198
199Need utils/parse_config.py
200
201======
202
203CDMS stuff is most of the mess
204==============================
205
2061. naToCdms.py holds:
207
208AbstractNAToCdms CLASS
209toCdmsFile
210createCdmsVariables
211toCdmsVariable
212createCdmsAuxVariables
213auxToCdmsVariable
214createCdmsAxes
215toCdmsAxes
216
2172. na2cdms.py:
218
219Command-line script
220
2213. bin/na2nc:
222
223Same as na2cdms.py ???
224
2254. cdms2na.py is the mother of all modules:
226
227compareAxes --> areAxesIdentical(a,b) cdms_utils
228compareVariables --> areDomainsIdentical(v1, v2) cdms_utils
229isAuxAndVar --> isAuxVarAndVar
230arrayToList utils
231listOfListsCreator utils
232getBestName cdms_utils - need some advice and compare with Dom
233getMissingValue cdms_utils
234fixHeaderLengthNowDefunct # Can destroy
235flatten2DTimeData aircraft
236modifyNADictCopy - needs a better name as it is specific
237cdms2na - 200 lines of code to do main conversion, needs to be split out into other stuff.
238 * getVariableCollections(f and varlist) --> (ordered_vars, other_vars)
239 * buildNADicts()
240 * writeToOutputFiles()
241
242class CdmsToNABuilder --> NAContentCollector: (naDict, varIDs, varBin)
243__init__ --> sets everything up and runs it move some to --> analyse()
244analyseVariables
245defineNAVars
246defineNAAuxVars
247getAxisDefinition
248defineNAGlobals
249defineNAComments
250defineGeneralHeader
251_useLocalRule --> Remove this and put it all in aircraft contrib bit
252
253=================
254
255naToCdms.py
256===========
257
258This is a sub-class of all NAFile objects. Bad idea. What we need is to:
259
260import convertor
261convertor.writeToNC(blah)
262convertor.convertToCdms(blah): (vars, global_atts)
263
264class NAToCdms
265
266toCdmsFile
267createCdmsVariables - does all
268toCdmsVariable - does each in turn
269CreateCdmsAuxVariables - does all
270auxToCdmsVariables - does each in turn
271createCdmsAxes - does all
272toCdmsAxes - does each in turn
273
274================
275
276Have I broken textParser.py's main function - is it same in old and new.
277
278=====
279
280NAFile2010:
281_readData[12] are crying out for useful names
282
283GREP
284====
285
286Need to do a lot of grepping for inconsistencies.
287
288The following need to be set to True or False (not yes,no):
289 * _normalizedX
290 * time_warning
291
292Global find and replace:
293
294floatFormat
295naDict
296=====================
297
298In naToCdms.py.NAToCdms.toCdmsAxis() there is a line naming the id (if too long) as:
299
300naAuxVariable....
301
302 - need to find where else this string is used and replace all with "naIndVariable" with map-back!
303
304===================
305
306Should we leave in the interactive time units checker in na_to_cdms.py - ask Charles
307
308=====
309cdms_map is not all done in the config file dict.
310
311Unit tests
312==========
313
3141001 appears to be done. Others are there as stubs but all need writing.
315
316We then need a set of nc_interface tests as well! Need NetCDF files to convert the other way as well. Need to be small.
317
318===============
319
320= Changes needed to make Nappy fit for purpose =
321
322
323 * Need error checking throughout - making it more robust but also checking on read will make Nappy a useful NASA Ames checker replacement.
324
325 * cdms interface should be in a separate part of the stack, not in core classes AND SUB-CLASSES
326
327 * Need to make an egg of it, but in the meantime we also need it installed as:
328
329   * nappy-0.2.3
330     * nappy
331     * bin
332     * nappy/nc_interface
333     * nappy/cdms_utils/
334     * nappy/utils
335     * nappy/na_file
336     * nappy/contrib/aircraft
337   
338
339 * GET REST FROM WIKI!
340
341
342===============
343localRules.py --> REMOVE COMPLETELY
344localRules/blah - remove this and put stuff in a top-level config file, or even remove altogether.
345
346localRules/aircraft.py - Need to consider how this can be a sub-class of Cdms2NA so that we push all the odd code into separate modules. Need to encapstulate the differences into one or two methods that are small and can be overridden.
347
348======
349bin/scanFAAM.py - put in contrib
350
351======
352
353version.py - put in config file.
354
355======
356
357general.py --> call it utils/xxxxx.py
358
359textParser --> utils/text_parser.py
360
361naError.py --> na_error/na_error.py
362
363naCore.py --> na_file/na_core.py
364
365listManipulator -_> utils/list_manipulator.py
366
367cdmsMap.py --> put in config file given simplicity
368
369Need utils/parse_config.py
370
371======
372
373CDMS stuff is most of the mess
374==============================
375
3761. naToCdms.py holds:
377
378AbstractNAToCdms CLASS
379toCdmsFile
380createCdmsVariables
381toCdmsVariable
382createCdmsAuxVariables
383auxToCdmsVariable
384createCdmsAxes
385toCdmsAxes
386
3872. na2cdms.py:
388
389Command-line script
390
3913. bin/na2nc:
392
393Same as na2cdms.py ???
394
3954. cdms2na.py is the mother of all modules:
396
397compareAxes --> areAxesIdentical(a,b) cdms_utils
398compareVariables --> areDomainsIdentical(v1, v2) cdms_utils
399isAuxAndVar --> isAuxVarAndVar
400arrayToList utils
401listOfListsCreator utils
402getBestName cdms_utils - need some advice and compare with Dom
403getMissingValue cdms_utils
404fixHeaderLengthNowDefunct # Can destroy
405flatten2DTimeData aircraft
406modifyNADictCopy - needs a better name as it is specific
407cdms2na - 200 lines of code to do main conversion, needs to be split out into other stuff.
408 * getVariableCollections(f and varlist) --> (ordered_vars, other_vars)
409 * buildNADicts()
410 * writeToOutputFiles()
411
412class CdmsToNABuilder --> NAContentCollector: (naDict, varIDs, varBin)
413__init__ --> sets everything up and runs it move some to --> analyse()
414analyseVariables
415defineNAVars
416defineNAAuxVars
417getAxisDefinition
418defineNAGlobals
419defineNAComments
420defineGeneralHeader
421_useLocalRule --> Remove this and put it all in aircraft contrib bit
422
423=================
424
425naToCdms.py
426===========
427
428This is a sub-class of all NAFile objects. Bad idea. What we need is to:
429
430import convertor
431convertor.writeToNC(blah)
432convertor.convertToCdms(blah): (vars, global_atts)
433
434class NAToCdms
435
436toCdmsFile
437createCdmsVariables - does all
438toCdmsVariable - does each in turn
439CreateCdmsAuxVariables - does all
440auxToCdmsVariables - does each in turn
441createCdmsAxes - does all
442toCdmsAxes - does each in turn
443
444================
445
446Have I broken textParser.py's main function - is it same in old and new.
447
448=====
449
450NAFile2010:
451_readData[12] are crying out for useful names
452
453GREP
454====
455
456Need to do a lot of grepping for inconsistencies.
457
458 * _normalizedX should be True|False not "yes","no"
459
460Global find and replace:
461
462floatFormat
463naDict
Note: See TracBrowser for help on using the repository browser.