Changeset 361


Ignore:
Timestamp:
30/07/15 10:37:13 (5 years ago)
Author:
mjuckes
Message:

more updates

Location:
CMIP6dreqbuild/trunk
Files:
1 added
7 edited
1 moved

Legend:

Unmodified
Added
Removed

  • CMIP6dreqbuild/trunk/docs/dreq.txt

    r356 r361  
    11##  
    22vocab var; MIP Variable; cmip.drv.001; def 
    3  - sn; units; description; procnote; procComment; prov; uuid 
     3 - sn {CF Standard Name} <externalTextRef>; units {Units}; description;  
     4 - procnote {Processing notes} <list> 
     5 - procComment {Processing comments} <freeText> 
     6 - prov {Provenance} <freeText> 
     7 - uuid <recordId> 
    48##  label==var 
    59vocab ovar; MIP Output Variable; cmip.drv.001; def 
    6  - uuid; comment; deflate_level; shuffle; ok_max_mean_abs; flag_meanings; type; ok_min_mean_abs; sn; deflate; valid_min; cell_methods; flag_values; cell_measures; out_name; modeling_realm; units; cell_methods_xx; valid_max; positive; mipTable; dimensions; vid; frequency 
    7 ## label==var, title==long name 
     10 - uuid <recordId> 
     11 - title {Long name} 
     12 - comment {Description} <freeText> 
     13 - deflate_level; shuffle; ok_max_mean_abs; flag_meanings; type; ok_min_mean_abs;  
     14 - sn {CF Standard Name} <externalTextRef> 
     15 - deflate; valid_min; cell_methods; flag_values; cell_measures; out_name; modeling_realm; units; cell_methods_xx; valid_max; positive; mipTable; dimensions; 
     16 - vid {Identifier for MIP Variable} <internalLink|var> 
     17 - frequency {Frequency}  <drsVocab> 
     18##  
    819vocab groupItem; Output Group Item; cmip.drv.001; def 
    9  - group;  table; freq; descriptionEx; shape; levels; tstyle; mask; misc; mip; uuid; new; gpid; vkey; vid 
    10 ## label==var, title==long name 
     20 - group;  table; freq; descriptionEx; shape; levels; tstyle; mask; misc; mip; uuid <recordId>; new;  
     21 - gpid {Identifier for request link} <internalLink>; vkey; 
     22 - vid {Identifier for MIP Variable} <internalLink|var> 
     23##  
    1124vocab revisedTabItem; Revised Table Item; cmip.drv.001; def 
    12  - table; mip; uuid; priority; vid 
    13 ##  label==tab, title??? 
     25 - table 
     26 - mip {Endorsed MIP requesting the variable} 
     27 - uuid <recordId>; priority;  
     28 - vid {Identifier for MIP Output Variable} <internalLink|ovar> 
     29 - vgid {Identifier for Variable Group} <internalLink|requestVarGroup> 
     30##   
    1431vocab requestVarGroup; Request link: linking a set of variables; cmip.drv.001; def 
    15  - uuid; mip; ref; refNote 
     32 - uuid <recordId> 
     33 - mip {Endorsed MIP requesting the data} 
     34 - ref; refNote 
    1635vocab requestLink; Request link: linking a set of variables and a set of experiments.; cmip.drv.001; def 
    17  - uuid; mip; tab; objective; grid; gridreq; comment; ref; refNote; refid 
    18 vocab requestItem; Request Item: specifying the nuber of years for an experiment; cmip.drv.001; def 
    19  - uuid; mip; tab; expt; rlid; ny 
     36 - uuid <recordId>; mip; tab; objective; grid; gridreq; comment; ref; refNote;  
     37 - refid {reference to a request Variable Group} <internalLink> 
     38vocab requestItem; Request Item: specifying the number of years for an experiment; cmip.drv.001; def 
     39 - uuid <recordId>; mip; tab; expt;  
     40 - rlid {Identifier of corresponding requestLink} <internalLink> 
     41 - ny [xs:integer] 
     42vocab remarks; Remarks about other items; cmip.drv.001; def 
     43 - uuid <recordId>;  
     44 - tid  {Target identifier: the record ID of the item this refers to.} <internalLink> 
     45 - tattr {Target attribute: an attribute of the target item, or 'ALL'} 
     46 - description {Free text remarks -- unless there are specific restrictions associated with the class attribute of this remark} 
     47 - class {Different classes of remark support a range of specialised usages: free (a free text comment), modelTypeReq (a model type requirement), modelTypeExcl (a model type exclusion), varAlt (an alternative variable that my be used instead of this one), varSup (an alternative variable which, if selected makes this one redundant), attChange (a change in the value of an attribute -- old value provided in 'techNote').} 
     48 - qid {Identifier required for classes modelTypeReq, modelTypeExcl, varAlt, varSup. For the first two classes, the identifier points to a model type specification, for the last two it points to an output variable specification.} 
     49 - techNote {Optional additional machine readable content (though not restricted by the schema)} 

  • CMIP6dreqbuild/trunk/docs/vocabFrameworkSchema.xsd

    r322 r361  
    1515  <xs:attribute name="title" type="xs:string"/> 
    1616  <xs:attribute name="type" type="xs:string"/> 
     17  <xs:attribute name="class" type="xs:string"/> 
     18  <xs:attribute name="techNote" type="xs:string"/> 
    1719</xs:complexType> 
    1820 

  • CMIP6dreqbuild/trunk/docs/xlst_xdsSchemaGen.xml

    r353 r361  
    7373         <xsl:apply-templates mode="main"/>   
    7474      </xs:all> 
    75     </xs:complexType> 
     75    </xs:complexType><xsl:text>&#xa;</xsl:text> 
     76  <xs:unique name="uniqueIdentifierConstraint"> 
     77  <xs:annotation> 
     78    <xs:appinfo>W3Schools Note</xs:appinfo> 
     79    <xs:documentation xml:lang="en"> 
     80       The uniqueIdentifierConstraint imposes the condition that the uuid attribute on "item" elements should have a unique value. 
     81    </xs:documentation> 
     82  </xs:annotation><xsl:text>&#xa;</xsl:text> 
     83    <xs:selector xpath=".//item"/> 
     84    <xs:field xpath="@uuid"/> 
     85  </xs:unique><xsl:text>&#xa;</xsl:text> 
    7686  </xs:element><xsl:text>&#xa;</xsl:text> 
    7787 

  • CMIP6dreqbuild/trunk/src/framework/dreq.py

    r353 r361  
    1515           self.mdInit( xmlMiniDom ) 
    1616 
    17  
    1817       def dictInit( self, dict ): 
    1918         for a in self.attributes: 
    … …  
    3130           else: 
    3231             self.__dict__[a] = self.defaults.get( a, self.globalDefault ) 
    33          
    3432 
    3533class config(object): 
    3634 
    3735  def __init__(self): 
    38     self.vdef = 'out/parVocabDefn.xml'  
     36    self.vdef = 'out/dreqDefn.xml'  
    3937    self.vsamp = 'out/vocabSample.xml' 
    4038    self.vsamp = '../workbook/trial.xml' 
    4139    self.vsamp = '../workbook/trial_20150724.xml' 
    4240    self.nts = collections.namedtuple( 'sectdef', ['tag','label','title','id','itemLabelMode'] ) 
    43     self.nti = collections.namedtuple( 'itemdef', ['tag','label','title','type'] ) 
     41    self.nti = collections.namedtuple( 'itemdef', ['tag','label','title','type','rClass','techNote'] ) 
    4442    self.ntt = collections.namedtuple( 'sect', ['header','attributes'] ) 
    4543 
    … …  
    127125      else: 
    128126        ty = "xs:string" 
    129       return self.nti( i.nodeName, l,t,ty ) 
     127      if i.hasAttribute( 'class' ): 
     128        cls = i.getAttribute( 'class' ) 
     129      else: 
     130        cls = None 
     131      if i.hasAttribute( 'techNote' ): 
     132        tn = i.getAttribute( 'techNote' ) 
     133      else: 
     134        tn = None 
     135      return self.nti( i.nodeName, l,t,ty,cls,tn ) 
    130136 
    131137c = config() 

  • CMIP6dreqbuild/trunk/src/framework/ptxt.py

    r360 r361  
    44ptxt.py [-f srcFile] [samp|defn] 
    55''' 
    6 import string, re, collections, sys 
     6import string, re, collections, sys, uuid 
    77 
    88if len(sys.argv) == 1: 
    … …  
    2727re_atdef = re.compile( '^(?P<n>\S+)\s*((\[(?P<b>.*?)\]|\{(?P<a>.*?)\}|\<(?P<c>.*?)\>)\s*){0,3}$' ) 
    2828 
    29 nt__itematt = collections.namedtuple( 'itematt', ['name','type','title','clss'] ) 
     29### tuble to define attributes of an item 
     30nt__itematt = collections.namedtuple( 'itematt', ['name','type','title','clss','techn'] ) 
    3031 
    3132vocab_elTmpl = '''<table label="%(label)s" title="%(title)s" id="%(id)s" itemLabelMode="%(ilm)s"> 
    … …  
    3334</table> 
    3435''' 
    35 ial_elTmpl = '  <rowAttribute label="%(label)s"%(wrappedType)s%(wrappedTitle)s%(wrappedClass)s/>' 
     36ial_elTmpl = '  <rowAttribute label="%(label)s"%(wrappedType)s%(wrappedTitle)s%(wrappedClass)s%(wrappedTechn)s/>' 
    3637 
    3738expl_Tmpl = '''<%(label)s class="vocab" title="%(title)s" id="%(id)s"> 
    … …  
    5455     self.kk = kk 
    5556     self.msg( '[%s] %s {%s:%s}' % (self.label, self.title, self.id, self.ilt) ) 
    56      self.itematts = [  nt__itematt( 'label','xs:string',None,None ), 
    57                         nt__itematt( 'title','xs:string',None,None ) ] 
     57     self.itematts = [  nt__itematt( 'label','xs:string',None,None,None ), 
     58                        nt__itematt( 'title','xs:string',None,None,None ) ] 
    5859 
    5960  def tmpl(self,oo=None,mode="defn"): 
    … …  
    6162       ss = [] 
    6263       for i in self.itematts: 
    63          print i 
    6464         if i.clss != None: 
    6565           wrappedClass = ' class="%s"' % i.clss 
    6666         else: 
    6767           wrappedClass = '' 
     68 
     69         if i.techn != None: 
     70           wrappedTechn = ' techNote="%s"' % i.techn 
     71         else: 
     72           wrappedTechn = '' 
    6873 
    6974         if i.title != None: 
    … …  
    102107           value = "noType" 
    103108           if i.type == "xs:string": 
    104              value = 'dummyAt' 
     109             if i.name == 'uuid': 
     110               value = str( uuid.uuid1() ) 
     111             else: 
     112               value = 'dummyAt' 
    105113           elif i.type == "xs:integer": 
    106114             value = '25' 
    … …  
    132140     bits = map( string.strip, string.split(line[3:], ';' ) ) 
    133141     for b in bits: 
    134        bn = string.split(b, ' ')[0] 
    135        x = self.pb(b) 
    136        self.itematts.append( nt__itematt( bn, x[0], x[1], x[2] ) ) 
    137        self.msg( '%s, %s, %s, %s' % (bn, x[0], x[1], x[2]) ) 
     142       if string.strip(b) != '': 
     143         bn = string.split(b, ' ')[0] 
     144         x = self.pb(b) 
     145         self.itematts.append( nt__itematt( bn, x[0], x[1], x[2], x[3] ) ) 
     146         self.msg( '%s, %s, %s, %s, %s' % (bn, x[0], x[1], x[2], x[3]) ) 
    138147 
    139148  def pb(self, b): 
    … …  
    151160      itemType = ee['b'] 
    152161      itemClass = ee['c'] 
     162      itemTechNote = None 
    153163      if itemType == None: 
    154164         itemType = 'xs:string' 
     165      if type(itemClass) == type('x'): 
     166        if string.find(itemClass, '|' ) != -1: 
     167          bits = string.split( itemClass, '|' ) 
     168          assert len(bits) == 2, 'ERROR.001.0001: failed to parse attribute definition: %s' % b 
     169          itemClass = bits[0] 
     170          itemTechNote = bits[1] 
    155171         
    156172    #if len(x) > 0: 
    … …  
    163179    #else: 
    164180      #itemType = 'xs:string' 
    165     return [itemType, itemTitle,itemClass]  
     181    return [itemType, itemTitle,itemClass,itemTechNote]  
    166182 
    167183class main(object): 

  • CMIP6dreqbuild/trunk/src/workbook/importShDreq.py

    r356 r361  
    11"""Import data request from shelves and put onto compliant XML document. 
    22------------------------------------------------------------------------ 
    3 A new uuid is generatd for elements of the "revisedtabitem" section/table (because it is not present in the shelve). 
     3A new uuid is generated for elements of the "revisedtabitem" section/table (because it is not present in the shelve). 
    44 
    55NOTES 
    … …  
    3434    self.src=src 
    3535    self.repl = collections.defaultdict( list ) 
     36    self.err0010 = collections.defaultdict( int ) 
    3637    self.replItems = {} 
    3738    self.importRepl() 
    … …  
    6465    print xsn 
    6566    print ff.keys() 
    66  
     67     
     68    self.eern = collections.defaultdict( list ) 
     69    self.eern2 = {} 
     70    ixrn = 5 
     71    dups = [] 
     72    for k in rq.rqvg.keys(): 
     73      if k[0] != '_': 
     74        kk = ['uid', 'mip', 'tab', 'objective', 'grid', 'gridreq', 'comment', 'ref', 'refNote', 'refid'] 
     75        kk = ['uuid', 'mip', 'label', 'title', 'ref', 'refNote'] 
     76        i = rq.rqvg[k] 
     77         
     78        self.eern[i[ixrn]].append( i[0] ) 
     79        k2 = '%s__%s' % (i[ixrn-1],i[ixrn]) 
     80        if self.eern2.has_key( k2 ): 
     81           dups.append( k2 ) 
     82        self.eern2[k2] = i[0] 
     83        assert i[0] == k, 'Bad key/uuid' 
     84 
     85    assert len( dups ) == 0, 'Duplicate refs: %s' % str(dups) 
     86         
    6787    for k in [ 'var','ovar','groupitem', 'revisedtabitem', 'requestlink', 'requestitem','requestvargroup' ]: 
    6888    ##for k in [ 'var']: 
    … …  
    165185## - table; mip; uuid; priority 
    166186            kk = ['var', 'table', 'mip', 'vid', 'priority'] 
     187            kvg = 'rev__%s.%s' % ( ll[2],ll[1] ) 
     188            if self.eern2.has_key(kvg): 
     189              vgid = self.eern2[kvg] 
     190            else: 
     191              vgid = '__none__' 
     192              self.err0010[kvg] += 1 
    167193            thisuuid = str( uuid.uuid1() ) 
    168194            thisl = string.strip(str( ll[0] ) ) 
    … …  
    172198            item.setAttribute( 'title', thisl ) 
    173199            item.setAttribute( 'uuid', thisuuid ) 
     200            item.setAttribute( 'vgid', vgid ) 
    174201            if self.insert.has_key(thisl): 
    175202                thisold = item.getAttribute( 'vid' ) 
    … …  
    193220            thisl = string.strip(str( '%s-%s-%s' % (ll[0],ll[1],ll[2]) ) ) 
    194221            for j in range(len(kk)): 
    195                 item.setAttribute( kk[j], str( ll[j] ) ) 
     222                if kk[j] == 'ny': 
     223                  thisv = str( int(ll[j]) ) 
     224                else: 
     225                  thisv = str( ll[j] ) 
     226                item.setAttribute( kk[j], thisv ) 
    196227            item.setAttribute( 'title', '%s, %s, %s' % (ll[0],ll[1],ll[2]) ) 
    197228            item.setAttribute( 'uuid', str(i) ) 
    … …  
    199230          elif k == 'requestvargroup': 
    200231            kk = ['uuid', 'mip', 'label', 'title', 'ref', 'refNote'] 
     232            thisl = string.replace( str( ll[2] ), '.', '-' ) 
    201233            for j in range(len(kk)): 
    202234                item.setAttribute( kk[j], str( ll[j] ) ) 
    … …  
    270302         self.insert[bits[1]] = bits[3] 
    271303 
    272 sampleXml = '../framework/out/parVocabSample.xml' 
     304sampleXml = '../framework/out/dreqSample.xml' 
    273305from scansh import rq 
    274306m = main(sampleXml, rq) 

  • CMIP6dreqbuild/trunk/srcMisc/scansh.py

    r356 r361  
    7474kmm = 0 
    7575knf = 0 
     76nn0 = 0 
     77nn1 = 0 
    7678for k in rq.grps.keys(): 
    7779  if k[0] != '_': 
    … …  
    7981    if r[-3][0] != '_': 
    8082      if r[-4] == 0: 
     83        nn0 += 1 
    8184        if rq.vars.has_key(r[-3]): 
    8285          nok1 += 1 
    … …  
    8588          knf += 1 
    8689      elif r[-4] == 1: 
     90        nn1 += 1 
    8791        if rq.rqvg.has_key(r[-3]): 
    8892          nok2 += 1 
    … …  
    9599      ##print '!GROUP: ',r 
    96100      km += 1 
    97 print 'CHECK.001.0004: on groupItems: %s,%s OK, %s missing reference, not found %s, bad "new" key %s ' % (nok1,nok2,km,knf,kmm) 
     101print 'CHECK.001.0004: on groupItems: %s,%s OK, %s missing reference, not found %s, bad "new" key %s [%s,%s] ' % (nok1,nok2,km,knf,kmm,nn0,nn1) 
    98102 
    99103nok1 = 0 

  • CMIP6dreqbuild/trunk/srcMisc/sx2.py

    r355 r361  
    419419for k in shrvg.keys(): 
    420420  if k[0] != '_': 
     421    assert not eervg.has_key( shrvg[k][2] ), 'Duplicate rvg: %s' % shrvg[k][2] 
    421422    eervg[shrvg[k][2]] = k 
    422423 
    … …  
    469470for t in mmmm: 
    470471  kmap000[t[0]] = t[1] 
     472 
     473## 
    471474## check links from variable output specs to group. 
    472475## need to identify what is meant by group labels in "Request Scoping" -- now in eervg.keys() 
    473476## 
     477## 29th July: loosin some links for groups of revised table items .... 
     478## 
     479shb = shelve.open( 'dreq_consol_tables_reviewed_b_v20150708', 'r' ) 
     480revisedTabKeys = shb.keys() 
     481shb.close() 
     482gpids = gpids + revisedTabKeys 
    474483ks = eervg.keys() 
    475484ks.sort() 
    … …  
    526535## 
    527536rvgref2 = collections.defaultdict( list ) 
     537lookup1a = collections.defaultdict( d1 ) 
     538lookup1b = {} 
    528539lookup1 = collections.defaultdict( d1 ) 
    529540lookup2 = collections.defaultdict( d1 ) 
    … …  
    532543sh['__info__'] = { 'label':'requestVarGroup', 'title':'Identify variable groups' } 
    533544sh['__cols__'] = [ 'uuid', 'mip', 'label','title','ref','refNote'] 
     545## C4MIP-OceanT2        C4MIP: C4MIP-OceanT2    C4MIP   new     C4MIP.C_Ocean_T2 
     546## label title mip ref refNote 
     547 
    534548#### 
    535549#### reconfigure to uuid, mip, label (no "."), title, class, refNote 
    … …  
    541555    id0 = shrvg[k][2] 
    542556    ttt,kkk = kmapk2[id0] 
    543     lookup1[ttt].d[kkk].append(k) 
     557    lookup1a[ttt].d[kkk].append(k) 
    544558    if ttt == 'new': 
    545559      rvgref2[kkk].append( shrvg[k][0] ) 
     560 
     561for k in lookup1a.keys(): 
     562  for k2 in lookup1a[k].d.keys(): 
     563    id0 = lookup1a[k].d[k2][0] 
     564    id =  str( uuid.uuid1() ) 
     565    lookup1[k].d[k2].append( id) 
     566    lookup1b[id0] = id 
     567  
     568ndup = 0 
     569for k in shlnks.keys(): 
     570  if k[0] != '_': 
     571    k1 = '%s.%s' % (shlnks[k][1],shlnks[k][2]) 
     572    assert kmapk2.has_key(k1), 'Key missing  %s, %s' % (k,k1) 
     573    if not kmapk2.has_key(k1): 
     574      print '###########',shlnks[k] 
     575    else: 
     576      ttt,kkk = kmapk2[k1] 
     577      id = lookup1[ttt].d[kkk][0] 
     578    if id == k: 
     579      print '***DUPLICATE:',k,k1, ttt, kkk, lookup1[ttt].d[kkk] 
     580      ndup += 1 
     581 
     582assert ndup==0, ndup 
    546583    ##sh[k] = list( shrvg[k] ) + list( kmapk2[id0] ) 
    547584    ##print '>>>>>',k,sh[k] 
    … …  
    549586  for k2 in lookup1[k].d.keys(): 
    550587    id = lookup1[k].d[k2][0] 
    551     id0 = shrvg[id][2] 
    552     mip = shrvg[id][1] 
     588    idi = lookup1a[k].d[k2][0] 
     589    id0 = shrvg[idi][2] 
     590    mip = shrvg[idi][1] 
    553591    ttt,kkk = kmapk2[id0] 
    554592    cls = k 
    … …  
    556594    refNote = k2 
    557595    if cls == 'new': 
    558       lab = sx2e.ngmap[kkk] 
     596      if kkk in revisedTabKeys: 
     597        lab = kkk 
     598        cls = 'rev' 
     599      else: 
     600        lab = sx2e.ngmap[kkk] 
    559601    elif cls == 'CMIP5': 
    560602      lab = string.replace( k2, '_', '-' ) 
    … …  
    568610    if sx2e.r1.match( lab ) == None: 
    569611      print 'NON_COMPLIANT label', lab 
    570       print k,k2,shrvg[id] 
    571       assert False 
     612      ## print k,k2,shrvg[id] 
     613      ## assert False 
    572614    sh[id] = [id,mip,lab,'%s: %s' % (mip,lab), cls, refNote] 
     615    print '>>>>>>>>>',sh[id] 
    573616 
    574617 
    … …  
    584627       
    585628sh.close() 
    586  
    587629 
    588630err001 = collections.defaultdict( int ) 
    … …  
    609651      r[-1] = '__none__' 
    610652    t = ee0_xref[k][0] 
    611     print '>>>>>>>>>>>>>>>>>>>>',t 
    612653    r += [t[0],t[1][0]] 
    613654    sh[k] = r[:] 
    … …  
    618659sh['__info__'] = { 'label':'requestLinks', 'title':'Links from variable groups to a request id' } 
    619660sh['__cols__'] = [ 'uid', 'mip', 'tab','objective','grid','gridreq','comment','ref','refNote','refid'] 
     661ndup = 0 
    620662for k in shlnks.keys(): 
    621663  if k[0] != '_': 
    … …  
    628670      id = lookup1[ttt].d[kkk][0] 
    629671      sh[k] = list( shlnks[k] ) + list( kmapk2[k1] ) + [id,] 
     672    if sh[k][0] == sh[k][-1]: 
     673      print 'DUPLICATE LINK:',k,sh[k] 
     674      ndup += 1 
     675print 'Numbr of duplicates %s (%s)' % (ndup, len(sh.keys())) 
    630676sh.close() 
    631677## link to group items is now via mip + group id, using stuff from kmapk2 .... 
Note: See TracChangeset for help on using the changeset viewer.