wiki:WorkingGrid/DataProviderSetup/PML/POLJan07

Version 14 (modified by pmiller, 13 years ago) (diff)

--

Peter's notes on NDG Meetings at POL 17-18 Jan. 2007

PML Actions

  • Peter - Test Bryan's Discovery search (available end Jan), then Browse GUI.
  • All - Fix PML DIF (then MOLES) records using Discovery search - must be done by end Feb for NERC Gateway.
  • Peter - Talk to Phil re cross-domain cookie advice for NEODAAS link-up.
  • Peter - circulate proposal for intelligent (Google) ordering of discovered records.
  • Mike - discuss with Andrew/Dom? how to progress 8-bit read methods.
  • Peter - discuss CSML2 Swath feasture with Andrew.

Project board

Issues

  • Progress: sluggish e.g. security, metadata production. Bryan says we should just go with the security development version if that works, even if this is a 'moving target'. Staffing problem with people getting dragged into other projects rather than NDG deliverables. Bryan says we should be driving the development now with science questions, which can now be asked at this stage, e.g. I want to compare these two datasets. Sue should be nastier about missed deadlines.
  • Staffing: Fabio may leave before end of NDG. Bryan says we should get no-cost extension, extend Andrew's work on CSML, ... Could be synchronisation issues between delayed components and DP effort.
  • DX: Steven Pascoe is re-engineering DX as a lightweight interface for DEWS, using mapplotter not CDAT. Ag has DX working for two feature types (overlay trajectory on plot), but must use Dom's CSML interface to hide the issues like a new CSML feature type for PML data files.

What is important to PML? We are doing OK with the metadata population (MOLES, DIF, CSML) - Sue says a week ago we jumped to 1,400 DIFs! (Bryan says granularity issues.) We need access to Browse interface to play with. Security is not as important as logging. We want to be able to extract/visualise our satellite data via CSML, DX, GeoSPLAT. If delivery via GeoSPLAT is problematic, what about OGC interfaces to WxS?

Lots of discussion (in Bryan's mindmap) on the technical questions that need resolving tomorrow, particularly regarding DX, CSML, ...

We will host services: Browse interface GUI, WCS- or DX-based backend, WCS- or DC-based frontend GUI (ie WMS Client or GeoSPLAT), ...

  • Metadata: Discussion on Roy's granularity issues, multiple related data entities for All Cruises->Individual cruises->CTD's, etc. Only the 'All Cruises' aggregated MOLES record will be discoverable as a DIF. Argh - Bryan hasn't implemented related entities in Browse (because he hasn't got a Stub-B schema...) so you can't actually get to the individual cruise yet! Suggested adding 'Discoverable' boolean parameter to each MOLES record.
  • I repeated my view that the DP deciding how to aggregate all their datasets is unlikely to be a successful approach, both in terms of getting a quality selection of DIFs from the DP, and in users being impressed with a Discovery search resulting in 'All cruises', 'All SeaWiFS data globally', etc., particularly if these have a gap for the time/region of interest. I believe all the MOLES records should have DIFs, and Discovery should do a better job like Google in indexing and ordering the results. E.g. if user searches 'chlor' then the aggregated datasets would be a sensible result, but if user searches 'chlor' + 1998 + North Sea then they should see a few matching cruises and a few closely matching satellite datasets (SeaWiFS North Sea 1998, SeaWiFS global 1998). Bryan thinks that this would be nice but won't be done in NDG2. I maintain there should be a relatively easy way to add this to Discovery, to avoid DP's having to mess with their datasets after they've gone to the trouble of releasing them. E.g. Are the DIF related record fields being used? Don't think so. Peter must write report if he wants this to happen in NDG2, and it must be really easy.
  • Risk review.
  • Issues review. The web-site integration with Dundee for NEODAAS I said is not an issue on NDG2 timescale. Even when the authentication system changes we can still keep the RSDAS website going for NDG purposes.

Continuation of board meeting

  • Documentation requirements: This was previously covered in the External Product Definitions mindmap, which needs Helen to prioritise and assign tasks.
  • See what QUEST are doing on QESA (Earth system atlas) - fancy website etc. Helen? go to QUEST open science meeting.
  • MOLES issues, if Kev is to leave project soon then his knowledge must be documented and transferred. There is a Stub-B schema in svn.

All-hands meeting

  • Rant from Sue about being behind on milestones.
  • Discovery website: demo of simple/advanced search. Revealed our records - 846 hits for temperature - immediate problems with data summary, lower case title, repository 'PIM Miller' and related record links. Need access to discovery search in order to debug these - Bryan says available on Glue by 25 Jan.
  • Security update (Phil): DEWS design. Using standard web services interfaces (e.g. Twisted) should be much more robust. Phil much happier with Beta, DP's like us should install that rather than continue with buggy Alpha. Something about role-mapping. Bryan suggests leaving Security out of the DX/etc. testing for few weeks, leave placeholders in code. Security Client stub code date? DP able to (re)install security date? Cross-domain cookie issues - talk to Phil for advice on NEODAAS.
  • Vocab server (Roy): new API v2 live bv end Feb.
  • CSML (Andrew): CSML2 powerpoint overview. Station name missing, or more general problem of passing relevant metadata through to CSML via 'metadata' tag, e.g. for finding and labelling in DX/vis stages. Documentation coming... though CSML1 doc was very comprehensive. DP's need to also talk with Dom for advice on CSML generation. Talk to Peter and Helen re Swath feature - e.g. I think the 'Time' value seen in Grid should be in Swath too, could be 'Reference time' - allows relationship to derived scenes but ambiguous, or 'Start/End? Times'; why not inherit Swath from Grid just adding extra fields?
  • Collaborations (Sue): MDIP deadlines approaching. NERC Metadata Gateway replaced by 1 March - so DP's must QC their records before then. EC INSPIRE - free access to data, final agreement allows Met Office to charge for certain data. Defra Climate Impacts Project (CDIP) (Ag).
  • Priorities given 20 weeks development time left (Bryan): run down on issues - Data services, Service bindings.
  • DX issues (Ag): DX will work by the end of NDG2, but needs 3 months of a programmer to help. Won't handle swaths. Bryan says separate the back/front ends. A backend between CSML and WxS may be straightforward, just has to getFeature. But Andrew is saying that is just delivering an XML document via WFS.
  • Andrew's proposal for differencing capability using WFS to get CSML then DataService? to get the relevent netCDFs. There must be missing parts of the interface that will need significant development.

Peter's proposal for search grouping

Hi Matthew,

I'm actually in a meeting right now, but briefly, I got through half of your paper on the train yesterday, and the rest on the way back tonight; sorry not to have done this sooner. I am very pleased to see that you have achieved real correlations with the frontal locations and that this forms a significant part of the story. I would certainly appreciate the opportunity to contribute as a co-author on this. The front maps I provided are not as yet a 'standard' NEODAAS (new name for RSDAS service) product or available elsewhere, and relies on my development of these techniques. So I view the front maps as a research contribution to your paper beyond the supply of SST data from NEODAAS; I believe I explained this at the start of your project.

I can also assist with improving the description of the front map methodology, and proof-reading the manuscript as there are several areas that are unclear or confusing: I'll do these as 'track changes' on the Word doc. There may be a repeating problem with your description of gradients in K units, whereas it should be K km-1; or maybe you are referring to the temperature difference across the whole front rather than a gradient. As the front mapping is a fairly novel technique I'd recommend showing a figure of one (or a few) 3-day map prior to the smoothed version; this is so the reader appreciates the characteristics of your data. I'll let you know if I've got further suggestions tomorrow.

Let me know whether this is all OK with you. Best regards, Peter