source: nappy/trunk/nappy/Plans for re-factoring 2008.txt @ 3352

Subversion URL: http://proj.badc.rl.ac.uk/svn/ndg/nappy/trunk/nappy/Plans for re-factoring 2008.txt@3353
Revision 3352, 12.2 KB checked in by astephen, 12 years ago (diff)

Got all versions so the re-factoring notes can be start of wiki page on changes made.

Line 
1= Changes needed to make Nappy fit for purpose =
2
3 * Need error checking throughout - making it more robust but also checking on read will make Nappy a useful NASA Ames checker replacement.
4
5 * cdms interface should be in a separate part of the stack, not in core classes AND SUB-CLASSES
6
7 * Need to make an egg of it, but in the meantime we also need it installed as:
8
9-------------------
10We need to decide what is required in terms of:
11
12    * actual formats
13    * mappings to/from NetCDF
14    * mapping to/from other formats (if required).
15
16    AS and CK have been analysing the required changes to make Nappy suitable for automated conversion of NetCDF-CF in DCIP to NASA Ames / CSV variant.
17
18        Here is an initial list of issues
19
20    * Put in the option to order variables explicitly, rather than relying on nasa_ames_var_number attributes.
21
22    * Support for 2110 – NX for the quickest changing IV needs to be converted to an auxiliary variable which does not fit nicely into the structure. In principle the second independent variable length changes at each first independent variable value but we may need it fixed.
23
24    * Refactoring to make it clearer and functions shorter but not too short.
25    * Add an option for the learning column.
26    * Proper use of FFI selection.
27    * In cdms2na.py, stopping filling an axis once the start and increment are known may not be enough. May need to populate all of the axis.
28    * Column headings for Excel users, this will clash with the current #End of normal comments# lines.
29    * Is it still right that the 4010 class is the same as 2010?
30    * Does comma separation option work correctly – any consequences for RDATE and DATE which retains some space separation?
31    * float rounding in nappy to get difference between independent variable values - and degree of accuracy (8.3f etc)
32    * cdms2na() needs argument "variable_order" that is a list of the order you want them to appear in 2110.
33    * Need sensible handling of rotated grid data where found (might be wrapper outside of nappy
34    * FFI 2110 is best for 2D columns for Excel etc. - Nappy must support this
35
36--------------------
37
38
39Dear Ag Stephans
40
41I found your nappy library for reading Ames Files at http://home.badc.rl.ac.uk/astephens/software/nappy/
42For one of my projects I want to use that lib in a plugin for a wiki (http://moinmo.in) to give some feedback about submitted/attached data files.
43
44Your web page states last modified at Wed, 06 Apr 2005 13:44:53 GMT
45 As I have seen on http://home.badc.rl.ac.uk/astephens/software/nappy/USAGE.txt
46"[NOTE: We plan to implement a getVariableArray(var_name) to grab a specific variable from the above array.]"
47
48I like to ask if there is probably a further version of that lib available?
49
50cheers
51Reimar Bauer
52
53----------------------
54
55
56=== Adding annotation column ===
57
58via argument: annotated=True|False
59
60Only on output. So only affects write methods.
61NOTE: could do read as well by reading in file and then removing column 1 (as long as we know the delimiter is a comma). But don't do this now.
62
63Ask Charles to come up with a list of definitions for each column in a config file.
64
65Need to defined names in config file and a ways of mapping to each of the lines by tagging to self.A, self.X self.XNAMES etc
66
67=== Adding CSV ===
68
69Via delimiter/spacer argument. Would be nice to have a writeCSV() method.
70
71Should 'csv' and 'delim' args be sent to write methods rather than __init__()?
72---
73
74===
75
76Do we need global DEBUG = True|False
77
78Then all the prints could be:
79
80if DEBUG == True: print "blah"
81
82===============
83localRules.py --> REMOVE COMPLETELY
84localRules/blah - remove this and put stuff in a top-level config file, or even remove altogether.
85
86localRules/aircraft.py - Need to consider how this can be a sub-class of Cdms2NA so that we push all the odd code into separate modules. Need to encapstulate the differences into one or two methods that are small and can be overridden.
87 * decided to dump all aircraft stuff in  an unsupported dir without refactoring in - it will probably never be used!
88
89=================
90
91import nappy  (via nappy_api.py module)
92nappy.convertNAToNC(na_file, nc_file)
93nappy.convertNCToNA(nc_file, na_file)
94
95================
96
97Have I broken textParser.py's main function - is it same in old and new - need a test for it!
98
99=====
100
101_readData[12] are crying out for useful names
102
103Should we leave in the interactive time units checker in na_to_cdms.py - ask Charles
104
105Unit tests
106==========
107
1081001 appears to be done. Others are there as stubs but all need writing.
109
110We then need a set of nc_interface tests as well! Need NetCDF files to convert the other way as well. Need to be small.
111
112
113-------
114
115= Changes needed to make Nappy fit for purpose =
116
117
118 * Need error checking throughout - making it more robust but also checking on read will make Nappy a useful NASA Ames checker replacement.
119
120 * cdms interface should be in a separate part of the stack, not in core classes AND SUB-CLASSES
121
122 * Need to make an egg of it, but in the meantime we also need it installed as:
123
124   * nappy-0.2.3
125     * nappy
126     * bin
127     * nappy/nc_interface
128     * nappy/cdms_utils/
129     * nappy/utils
130     * nappy/na_file
131     * nappy/contrib/aircraft
132   
133
134 * GET REST FROM WIKI!
135
136
137===============
138localRules.py --> REMOVE COMPLETELY
139localRules/blah - remove this and put stuff in a top-level config file, or even remove altogether.
140
141localRules/aircraft.py - Need to consider how this can be a sub-class of Cdms2NA so that we push all the odd code into separate modules. Need to encapstulate the differences into one or two methods that are small and can be overridden.
142
143======
144bin/scanFAAM.py - put in contrib
145
146======
147
148version.py - put in config file.
149
150======
151
152general.py --> call it utils/xxxxx.py
153
154textParser --> utils/text_parser.py
155
156naError.py --> na_error/na_error.py
157
158naCore.py --> na_file/na_core.py
159
160listManipulator -_> utils/list_manipulator.py
161
162cdmsMap.py --> put in config file given simplicity
163
164Need utils/parse_config.py
165
166======
167
168CDMS stuff is most of the mess
169==============================
170
1711. naToCdms.py holds:
172
173AbstractNAToCdms CLASS
174toCdmsFile
175createCdmsVariables
176toCdmsVariable
177createCdmsAuxVariables
178auxToCdmsVariable
179createCdmsAxes
180toCdmsAxes
181
1822. na2cdms.py:
183
184Command-line script
185
1863. bin/na2nc:
187
188Same as na2cdms.py ???
189
1904. cdms2na.py is the mother of all modules:
191
192compareAxes --> areAxesIdentical(a,b) cdms_utils
193compareVariables --> areDomainsIdentical(v1, v2) cdms_utils
194isAuxAndVar --> isAuxVarAndVar
195arrayToList utils
196listOfListsCreator utils
197getBestName cdms_utils - need some advice and compare with Dom
198getMissingValue cdms_utils
199fixHeaderLengthNowDefunct # Can destroy
200flatten2DTimeData aircraft
201modifyNADictCopy - needs a better name as it is specific
202cdms2na - 200 lines of code to do main conversion, needs to be split out into other stuff.
203 * getVariableCollections(f and varlist) --> (ordered_vars, other_vars)
204 * buildNADicts()
205 * writeToOutputFiles()
206
207class CdmsToNABuilder --> NAContentCollector: (naDict, varIDs, varBin)
208__init__ --> sets everything up and runs it move some to --> analyse()
209analyseVariables
210defineNAVars
211defineNAAuxVars
212getAxisDefinition
213defineNAGlobals
214defineNAComments
215defineGeneralHeader
216_useLocalRule --> Remove this and put it all in aircraft contrib bit
217
218=================
219
220naToCdms.py
221===========
222
223This is a sub-class of all NAFile objects. Bad idea. What we need is to:
224
225import convertor
226convertor.writeToNC(blah)
227convertor.convertToCdms(blah): (vars, global_atts)
228
229class NAToCdms
230
231toCdmsFile
232createCdmsVariables - does all
233toCdmsVariable - does each in turn
234CreateCdmsAuxVariables - does all
235auxToCdmsVariables - does each in turn
236createCdmsAxes - does all
237toCdmsAxes - does each in turn
238
239================
240
241Have I broken textParser.py's main function - is it same in old and new.
242
243=====
244
245NAFile2010:
246_readData[12] are crying out for useful names
247
248GREP
249====
250
251Need to do a lot of grepping for inconsistencies.
252
253The following need to be set to True or False (not yes,no):
254 * _normalizedX
255 * time_warning
256
257Global find and replace:
258
259floatFormat
260naDict
261=====================
262
263In naToCdms.py.NAToCdms.toCdmsAxis() there is a line naming the id (if too long) as:
264
265naAuxVariable....
266
267 - need to find where else this string is used and replace all with "naIndVariable" with map-back!
268
269===================
270
271Should we leave in the interactive time units checker in na_to_cdms.py - ask Charles
272
273=====
274cdms_map is not all done in the config file dict.
275
276Unit tests
277==========
278
2791001 appears to be done. Others are there as stubs but all need writing.
280
281We then need a set of nc_interface tests as well! Need NetCDF files to convert the other way as well. Need to be small.
282
283===============
284
285= Changes needed to make Nappy fit for purpose =
286
287
288 * Need error checking throughout - making it more robust but also checking on read will make Nappy a useful NASA Ames checker replacement.
289
290 * cdms interface should be in a separate part of the stack, not in core classes AND SUB-CLASSES
291
292 * Need to make an egg of it, but in the meantime we also need it installed as:
293
294   * nappy-0.2.3
295     * nappy
296     * bin
297     * nappy/nc_interface
298     * nappy/cdms_utils/
299     * nappy/utils
300     * nappy/na_file
301     * nappy/contrib/aircraft
302   
303
304 * GET REST FROM WIKI!
305
306
307===============
308localRules.py --> REMOVE COMPLETELY
309localRules/blah - remove this and put stuff in a top-level config file, or even remove altogether.
310
311localRules/aircraft.py - Need to consider how this can be a sub-class of Cdms2NA so that we push all the odd code into separate modules. Need to encapstulate the differences into one or two methods that are small and can be overridden.
312
313======
314bin/scanFAAM.py - put in contrib
315
316======
317
318version.py - put in config file.
319
320======
321
322general.py --> call it utils/xxxxx.py
323
324textParser --> utils/text_parser.py
325
326naError.py --> na_error/na_error.py
327
328naCore.py --> na_file/na_core.py
329
330listManipulator -_> utils/list_manipulator.py
331
332cdmsMap.py --> put in config file given simplicity
333
334Need utils/parse_config.py
335
336======
337
338CDMS stuff is most of the mess
339==============================
340
3411. naToCdms.py holds:
342
343AbstractNAToCdms CLASS
344toCdmsFile
345createCdmsVariables
346toCdmsVariable
347createCdmsAuxVariables
348auxToCdmsVariable
349createCdmsAxes
350toCdmsAxes
351
3522. na2cdms.py:
353
354Command-line script
355
3563. bin/na2nc:
357
358Same as na2cdms.py ???
359
3604. cdms2na.py is the mother of all modules:
361
362compareAxes --> areAxesIdentical(a,b) cdms_utils
363compareVariables --> areDomainsIdentical(v1, v2) cdms_utils
364isAuxAndVar --> isAuxVarAndVar
365arrayToList utils
366listOfListsCreator utils
367getBestName cdms_utils - need some advice and compare with Dom
368getMissingValue cdms_utils
369fixHeaderLengthNowDefunct # Can destroy
370flatten2DTimeData aircraft
371modifyNADictCopy - needs a better name as it is specific
372cdms2na - 200 lines of code to do main conversion, needs to be split out into other stuff.
373 * getVariableCollections(f and varlist) --> (ordered_vars, other_vars)
374 * buildNADicts()
375 * writeToOutputFiles()
376
377class CdmsToNABuilder --> NAContentCollector: (naDict, varIDs, varBin)
378__init__ --> sets everything up and runs it move some to --> analyse()
379analyseVariables
380defineNAVars
381defineNAAuxVars
382getAxisDefinition
383defineNAGlobals
384defineNAComments
385defineGeneralHeader
386_useLocalRule --> Remove this and put it all in aircraft contrib bit
387
388=================
389
390naToCdms.py
391===========
392
393This is a sub-class of all NAFile objects. Bad idea. What we need is to:
394
395import convertor
396convertor.writeToNC(blah)
397convertor.convertToCdms(blah): (vars, global_atts)
398
399class NAToCdms
400
401toCdmsFile
402createCdmsVariables - does all
403toCdmsVariable - does each in turn
404CreateCdmsAuxVariables - does all
405auxToCdmsVariables - does each in turn
406createCdmsAxes - does all
407toCdmsAxes - does each in turn
408
409================
410
411Have I broken textParser.py's main function - is it same in old and new.
412
413=====
414
415NAFile2010:
416_readData[12] are crying out for useful names
417
418GREP
419====
420
421Need to do a lot of grepping for inconsistencies.
422
423 * _normalizedX should be True|False not "yes","no"
424
425Global find and replace:
426
427floatFormat
428naDict
Note: See TracBrowser for help on using the repository browser.