Working Grid


The NDG working grid, with dataset population and deployed NDG services at all partners. As much data is as a practicable will be available via NDG at each provider (it is recognised that major effort is required to populate NDG metadata, and it will not be possible for all datasets to be NDG-enabled within the NDG2 timeframe).

Product purpose

To improve discovery, interpretation, and usage of data across discipline boundaries.

Product components

  1. Deployed central services
  2. Roles defined and role mappings deployed at all partners.
  3. Firewall problems resolved at all partners.
  4. Data provider services deployed at all partners
  5. Data and metadata populating the NDG information infrastructure at all partners for as much data is as practicable.
  6. Security services deployed and operational at all partners - see WGSecurity subcomponent

Key Integration Milestones

  1. Implementation One: install databases, and populate metadata instances (January 31, 2006) including OAI.
  2. Feedback One: (Feb 2006)
  3. Implementation Two: pseudo-grid (May, 2006)
  4. Feedback Two
  5. NDG Alpha (June 2006): Running grid (many missing services, but discovery and security must be running)
  6. Implementation Three: August 2006
  7. Feedback Three
  8. Implementation Four: as much as possible (September 2006)
  9. Feedback Four
  10. Data Checkpoint (how much data is installed, what’s stopping more data?)
  11. NDG Beta (January 2007): Running grid (all services present)

Integration Dependencies

  1. Attribute Authority (T12_Security)
  2. Session Manager (T12_Security)
  3. DX-Web Service Interface (T03_DataExtractor)
  4. Delivery Service (T05_Delivery)
  5. CSML Wrapper Software to read CSML and create NetCDF (T02_CSML)
  6. CSML Parser (T02_CSML)
  7. Geosplat Web Service Interface (T04_Geosplat)

Team members

Core people:

  • BODC = VPK/RKL (+Michael Hughes?)

Internal Development Stages


Due by/Milestone: Implementation 1/Feedback 1, Feb 2006


  1. [Definite] Install database package, including OAI
    1. new webserver (RHEL)
    2. reinstall eXist and OAI clients
  2. [Definite] Populate MOLES instances: partially complete, MOLES operational, more population required
    1. add Obs Stn and DP Tools to MOLES records
    2. begin generating metadata for majority of RSDAS holdings e.g. per active region per sensor.
  3. [Discuss] MOLES modifications (SensorML?)
  4. [Definite] Define local security roles


  1. [Definite] Install database package, including OAI - complete, updating daily
  2. [Definite] Populate MOLES instances see comment on MOLES modifications
  3. [Discuss] MOLES modifications
    1. Need to modify database to comply with MOLES structure (want to study what BODC did)
    2. Still waiting for model data from BODC (19/Apr/06)
  4. [Definite] Define local security roles


  1. [Definite] Install database package, including OAI
    1. Installed tomcat version of exist (19/Apr/06)
  2. [Definite] Populate MOLES instances - complete, 200 MOLES records removed and replaced with 45 with complete deployments, people+organisation records also generated
  3. [Discuss] MOLES modifications
    1. Removed all the attributes from the <dgMetadata> element in all the Moles records, to run against Kev’s new XQUERY
  4. [Definite] Populate a couple of CSML Instances
    1. In progress, working on a manually mapped point series feature
  5. [Definite] Define local security roles


  1. [Definite] Install database package, including OAI
  2. [Definite] Populate MOLES instances
    1. Complete, anything further is waiting on new BADC catalogue based on MOLES
  3. [Discuss] MOLES modifications
    1. Some issues raised with Kev as new BADC catalogue based on MOLES is implemented
  4. [Definite] Populate a couple of CSML Instances
    1. Some populated, but not sure if they correspond to any existing BADC datasets. Finding out from DL
  5. [Definite] Define local security roles


Due by/Milestone: Implementation 2/Feedback 2, May 2006


  1. [Definite] Get new webserver actually working
    1. Substantial job due to legacy code base
  2. [Definite] Local UserDB needed, coupled to roles
  3. [Definite] Establish which datasets are to be available and when
  4. [Definite] MOLES catalogue for instances goes live (with test user)
  5. [Definite] OAI scripts running from MOLES to OAI harvester
    1. Complete, but stability issues with OAI provider (fixed)
  6. [Discuss] Security: facilitate Phil installing and testing security system at PML.
  7. [Discuss] Data Provider Requirements:
    1. communicate issues with WG team, e.g. lack of documentation.
  8. [Definite] Integrate NDG authentication with RSDAS authentication system for data access.
  9. [Definite] Populate a couple of CSML Instances.
  10. [Discuss] CSML issues


  1. [Definite] Install initial security system at NOCS (with Phil)
  2. [Probable] Connect local userDB to NDG attribute authority
    1. Discussing and trialling solutions with Phil (Mar 06)
    2. Security client software working, now trying connection with NOCS database (19/Apr/06)
  3. [Definite] MOLES database goes live
  4. [Definite] OAI scripts running from MOLES to OAI harvester
  5. [Definite] Populate a couple of CSML Instances
    1. Waiting on OCCAM data (Mar 06)
    2. Still waiting (19/Apr/06)
  6. [Discuss] CSML issues
  7. [Probable] Consider caching requirements


  1. [Definite] Instantiate NDG attribute authority
    1. Instantiated, needs testing
  2. [Definite] Populate MOLES instances (which ones?)
    1. Regenerate Moles with Deployment records (including security information)
    2. Automate the BODC Moles generation software. (already around 45 datasets with deployments)
  3. [Discuss] Populate CSML instances (which ones?)
    1. Handcode 1 sea-level (points), 1 CTD (profile) instance with complete MOLES and CSML
  4. [Discuss] CSML issues


  1. [Definite] Instantiate NDG attribute authority
    1. Prototype installed and tested, working from bodc-to-badc, but problems badc-to-bodc. Full ndg aa probably not possible for may, more like alpha. (Mar 06)
    2. Works internally, waiting on solution to cross-domain cookies before can be tested externally (ticket:41) (19/Apr/06)
  2. [Definite] Populate MOLES instances (which ones?)
    1. Dependent on new badc catalogue based on moles being set up and populated. Could populate small number of specific datasets by hand if required for corresponding to csml instances. (Mar 06)
    2. Populated, but not live. Could start writing MOLES extraction (resources) (19/Apr/06)
  3. [Discuss] Populate CSML instances (which ones?)
    1. Decided list of datasets to-do. DL working on scanner which should enable more automation, at least with netcdf based data.
    2. Some populated, need to tie-in with MOLES instances (may not match up currently) (19/Apr/06)
  4. [Discuss] CSML issues
    1. Finding out from DL (Mar 06)
    2. Meeting with Dom booked (19/Apr/06)


Due by/Milestone: NDG Alpha (June 2006)

Various / All groups

  1. [Definite] Discovery services working
    1. Created list of minimum things to-do to make it ‘production’. That is, good enough to take over from NERC Metadata Gateway.
    2. Done, but bounced back (milestone)
  2. [Definite] Sites providing A and B data, listed and working on portal
  3. [Definite] OAI harvesting automated at NDG portal
    1. About 75% complete (Mar 06)
    2. 90% complete (19/Apr/06)
  4. [Definite] Security infrastructure in place, but not necessarily in use
  5. [Definite] Demonstrate a successful role mapping (get an attribute certificate)
    1. BODC has most of the resources supplied and available to run the NDG Security API (March 06)
    2. Cross domain cookies issue again
  6. [Discuss] Demonstrate secured application (DX? e.g. non-BADC user able to use the DX in some way)
  7. [Definite] Feedback to Data Provider package regarding installation instructions, etc


Due by/Milestone: External Product Definition

Mostly a checkpoint to see if there are any problems before PreBeta? and make sure information is available and disseminated.

Various / All groups

  1. [Discuss] Establish conformance/testing procedures for releases
  2. [Discuss] Feedback to documentation team
  3. [Discuss] Quantify available datasets and issues with further population (mini data checkpoint)
  4. [Discuss] Security infrastructure operational for most services
  5. [Discuss] Secure MOLES browsing working
  6. [Discuss] Test installs of (secured) Data Delivery service to a few DPs?
  7. [Discuss] Information on CSML (v2) updates available to DPs


Due by/Milestone: PreBeta

This stage is intended to try and pull together the back-end services that make Beta possible.

Various / All groups

  1. [Discuss] Discovery portal and harvesting operational
  2. [Discuss] Deployment of backend services to DPs
    1. DX backend (web-services) (cf. ticket:82)
    2. Delivery service install at DPs
    3. Security updates
  3. [Discuss] Security role-mapping between DPs complete (wiki:RoleMapping)
  4. [Discuss] Ensure integration of security into NDG "applications"
    1. Delivery, DX
    2. DPs (login, auth)
    3. Browse, GeoSPlAT
  5. [Discuss] CSML I/O layer working for DP-specific formats
    1. BADC (ticket?)
    2. BODC (ticket:186)
    3. NOCS (ticket:187)
    4. PML (ticket?)
  6. [Discuss] MOLES + CSML population (inc security) checkpoint (wiki:CSML_Population)


Due by/Milestone: Beta

Various / All groups

  1. [Discuss] GeoSPlAT deployment
    1. DX testing (ticket:249)
    2. GS web frontend (ticket:93)
  2. [Discuss] CSML services (opendap, W*S) available?
  3. [Discuss] MOLES + CSML population checkpoint (wiki:CSML_Population)
  4. [Discuss] Discovery: web services
  5. [Discuss] DataProvider/ClientPackage?: python install preparatory for packaging (ticket:300)
  6. [Definite] All services working and some data available(!)


Due by/Milestone: PostBeta_review

This stage is a review of how we did for beta, what needs doing, and making a public release.

  1. [Discuss] Discovery (and user-visible components): user testing
  2. [Discuss] Security system test plan & testing
  3. [Discuss] General web services testing plan
  4. [Discuss] Documentation plan? (depends on ticket:290)
  5. [Discuss] DataProvider: packaging up a release
  6. [Discuss] Public release?


Due by/Milestone: PROD

  1. [Discuss] MOLES complete
  2. [Discuss] CSML complete
  3. [Discuss] Security fully operational (system in place for updates and new roles?)
  4. [Discuss] CSML services, DX, GeoSPlAT, delivery working
  5. [Discuss] Discovery portal and harvesting operational
  6. [Discuss] Documentation available (promotional, DP, user, developer)
  7. [Discuss] Future planning (maintenance, etc)

Notes for discussion (temporary while planning/updating TiD)

Not sure about where these fit:

  • Logging?
  • Will GeoSPlAT be locally installed at DPs?
  • Need to integrate browse into plan, but not sure which package it belongs in (moles? discovery?)
  • Vocabulary server usage (part of metadata population, do we need this to be a WG integration task?)

Other comments:

  • Large time gap between Alpha and next checkpoint, might want to fill this in more