wiki:WorkingGrid/DataProviderSetup/PML/POLJan07

Peter's notes on NDG Meetings at POL 17-18 Jan. 2007

PML Actions

  • Peter - Test Bryan's Discovery search (available end Jan), then Browse GUI.
  • All - Fix PML DIF (then MOLES) records using Discovery search - must be done by end Feb for NERC Gateway.
  • Peter - Talk to Phil re cross-domain cookie advice for NEODAAS link-up.
  • Peter - circulate proposal for intelligent (Google) ordering of discovered records.
  • Mike - discuss with Andrew/Dom? how to progress 8-bit read methods.
  • Peter - discuss CSML2 Swath feature with Andrew.
  • Peter - contribute to DX GUI design team with Helen, Fabio, ...

Project board

Issues

  • Progress: sluggish e.g. security, metadata production. Bryan says we should just go with the security development version if that works, even if this is a 'moving target'. Staffing problem with people getting dragged into other projects rather than NDG deliverables. Bryan says we should be driving the development now with science questions, which can now be asked at this stage, e.g. I want to compare these two datasets. Sue should be nastier about missed deadlines.
  • Staffing: Fabio may leave before end of NDG. Bryan says we should get no-cost extension, extend Andrew's work on CSML, ... Could be synchronisation issues between delayed components and DP effort.
  • DX: Steven Pascoe is re-engineering DX as a lightweight interface for DEWS, using mapplotter not CDAT. Ag has DX working for two feature types (overlay trajectory on plot), but must use Dom's CSML interface to hide the issues like a new CSML feature type for PML data files.

What is important to PML? We are doing OK with the metadata population (MOLES, DIF, CSML) - Sue says a week ago we jumped to 1,400 DIFs! (Bryan says granularity issues.) We need access to Browse interface to play with. Security is not as important as logging. We want to be able to extract/visualise our satellite data via CSML, DX, GeoSPLAT. If delivery via GeoSPLAT is problematic, what about OGC interfaces to WxS?

Lots of discussion (in Bryan's mindmap) on the technical questions that need resolving tomorrow, particularly regarding DX, CSML, ...

We will host services: Browse interface GUI, WCS- or DX-based backend, WCS- or DC-based frontend GUI (ie WMS Client or GeoSPLAT), ...

  • Metadata: Discussion on Roy's granularity issues, multiple related data entities for All Cruises->Individual cruises->CTD's, etc. Only the 'All Cruises' aggregated MOLES record will be discoverable as a DIF. Argh - Bryan hasn't implemented related entities in Browse (because he hasn't got a Stub-B schema...) so you can't actually get to the individual cruise yet! Suggested adding 'Discoverable' boolean parameter to each MOLES record.
  • I repeated my view that the DP deciding how to aggregate all their datasets is unlikely to be a successful approach, both in terms of getting a quality selection of DIFs from the DP, and in users being impressed with a Discovery search resulting in 'All cruises', 'All SeaWiFS data globally', etc., particularly if these have a gap for the time/region of interest. I believe all the MOLES records should have DIFs, and Discovery should do a better job like Google in indexing and ordering the results. E.g. if user searches 'chlor' then the aggregated datasets would be a sensible result, but if user searches 'chlor' + 1998 + North Sea then they should see a few matching cruises and a few closely matching satellite datasets (SeaWiFS North Sea 1998, SeaWiFS global 1998). Bryan thinks that this would be nice but won't be done in NDG2. I maintain there should be a relatively easy way to add this to Discovery, to avoid DP's having to mess with their datasets after they've gone to the trouble of releasing them. E.g. Are the DIF related record fields being used? Don't think so. Peter must write report if he wants this to happen in NDG2, and it must be really easy.
  • Risk review.
  • Issues review. The web-site integration with Dundee for NEODAAS I said is not an issue on NDG2 timescale. Even when the authentication system changes we can still keep the RSDAS website going for NDG purposes.

Continuation of board meeting

  • Documentation requirements: This was previously covered in the External Product Definitions mindmap, which needs Helen to prioritise and assign tasks.
  • See what QUEST are doing on QESA (Earth system atlas) - fancy website etc. Helen? go to QUEST open science meeting.
  • MOLES issues, if Kev is to leave project soon then his knowledge must be documented and transferred. There is a Stub-B schema in svn.

All-hands meeting

  • Rant from Sue about being behind on milestones.
  • Discovery website: demo of simple/advanced search. Revealed our records - 846 hits for temperature - immediate problems with data summary, lower case title, repository 'PIM Miller' and related record links. Need access to discovery search in order to debug these - Bryan says available on Glue by 25 Jan.
  • Security update (Phil): DEWS design. Using standard web services interfaces (e.g. Twisted) should be much more robust. Phil much happier with Beta, DP's like us should install that rather than continue with buggy Alpha. Something about role-mapping. Bryan suggests leaving Security out of the DX/etc. testing for few weeks, leave placeholders in code. Security Client stub code date? DP able to (re)install security date? Cross-domain cookie issues - talk to Phil for advice on NEODAAS.
  • Vocab server (Roy): new API v2 live bv end Feb.
  • CSML (Andrew): CSML2 powerpoint overview. Station name missing, or more general problem of passing relevant metadata through to CSML via 'metadata' tag, e.g. for finding and labelling in DX/vis stages. Documentation coming... though CSML1 doc was very comprehensive. DP's need to also talk with Dom for advice on CSML generation. Talk to Peter and Helen re Swath feature - e.g. I think the 'Time' value seen in Grid should be in Swath too, could be 'Reference time' - allows relationship to derived scenes but ambiguous, or 'Start/End? Times'; why not inherit Swath from Grid just adding extra fields?
  • Collaborations (Sue): MDIP deadlines approaching. NERC Metadata Gateway replaced by 1 March - so DP's must QC their records before then. EC INSPIRE - free access to data, final agreement allows Met Office to charge for certain data. Defra Climate Impacts Project (CDIP) (Ag).
  • Priorities given 20 weeks development time left (Bryan): run down on issues - Data services, Service bindings.
  • DX issues (Ag): DX will work by the end of NDG2, but needs 3 months of a programmer to help. Won't handle swaths. Bryan says separate the back/front ends. A backend between CSML and WxS may be straightforward, just has to getFeature. But Andrew is saying that is just delivering an XML document via WFS.
  • Andrew's proposal for differencing capability using WFS to get CSML then DataService? to get the relevent netCDFs. There must be missing parts of the interface that will need significant development.
  • DX GUI: What does it look like? NOT the GeoSPLAT GUI and visualisation part which is OK, but the GUI which allows the user to decide which CSML features they want to visualise/combine, calls the back-end data services and generates an ouput netCDF (which is passed to GeoSPLAT or WxS). I must keep in mind the boundaries between these different services. Sue to convene GUI design team (me, Helen, Fabio to write gui?) which will explore radical designs based on CSML features (probably using WFS services). E.g. differencing, multiple data centres, trancon-type cruise-track coverage on SST map, compare CTD section against 3D model section. I said this should start from the DP Use Cases previously provided.
  • NDG Identifiers (Bryan): read wiki:Identifiers. Use double underscore, change our DIF to his recommended format. DX will take at most identifiers for two CSML feature types.
  • Software tools required at DP's: include Twisted, FastCGI (for the WSGI services) - may need Apache changes.
  • Timetabling for next period: Only one for PML: GUI team make decisions and decision by end Feb - storyboard interface, maybe via AG. Then Fabio building GUI March-May. Also improve metadata content. PML deploy Browse Bryan agrees with me that we need a simple end-to-end prototype working all the time (on Glue), so PML can check that their EO maps discover/browse/dx/geosplat. NDG Beta will be about browse, discovery and security.

Peter's proposal for search grouping