wiki:T10_WorkingGrid

Version 10 (modified by mggr, 13 years ago) (diff)

think I have all issues now

Working Grid

Description

The NDG working grid, with dataset population and deployed NDG services at all partners. As much data is as a practicable will be available via NDG at each provider (it is recognised that major effort is required to populate NDG metadata, and it will not be possible for all datasets to be NDG-enabled within the NDG2 timeframe).

Product purpose

To improve discovery, interpretation, and usage of data across discipline boundaries.

Product components

  1. Deployed central services
  2. Roles defined and role mappings deployed at all partners.
  3. Firewall problems resolved at all partners.
  4. Data provider services deployed at all partners
  5. Data and metadata populating the NDG information infrastructure at all partners for as much data is as practicable.
  6. Security services deployed and operational at all partners - see WGSecurity subcomponent

Key Integration Milestones

  1. Implementation One: install databases, and populate metadata instances (January 31, 2006) including OAI.
  2. Feedback One: (Feb 2006)
  3. Implementation Two: pseudo-grid (May, 2006)
  4. Feedback Two
  5. NDG Alpha (June 2006): Running grid (many missing services, but discovery and security must be running)
  6. Implementation Three: August 2006
  7. Feedback Three
  8. Implementation Four: as much as possible (September 2006)
  9. Feedback Four
  10. Data Checkpoint (how much data is installed, what’s stopping more data?)
  11. NDG Beta (January 2007): Running grid (all services present)

Integration Dependencies

  1. Attribute Authority (T12_Security)
  2. Session Manager (T12_Security)
  3. DX-Web Service Interface (T03_DataExtractor)
  4. Delivery Service (T05_Delivery)
  5. CSML Wrapper Software to read CSML and create NetCDF (T02_CSML)
  6. CSML Parser (T02_CSML)
  7. Geosplat Web Service Interface (T04_Geosplat)

Team members

Core people:

  • PML = MGG/PIM
  • NOCS = FV/HNS
  • BODC = VPK/RKL (+Michael Hughes?)
  • BADC = SEL/BNL

Internal Development Stages

Grid-V01

Due by/Milestone: Implementation 1/Feedback 1, Feb 2006

PML

  1. [Definite] Install database package, including OAI
    1. new webserver (RHEL)
    2. reinstall eXist and OAI clients
  2. [Definite] Populate MOLES instances: partially complete, MOLES operational, more population required
    1. add Obs Stn and DP Tools to MOLES records
    2. begin generating metadata for majority of RSDAS holdings e.g. per active region per sensor.
  3. [Discuss] MOLES modifications (SensorML?)
  4. [Definite] Define local security roles

NOCS

  1. [Definite] Install database package, including OAI - complete, updating daily
  2. [Definite] Populate MOLES instances see comment on MOLES modifications
  3. [Discuss] MOLES modifications
    1. Need to modify database to comply with MOLES structure (want to study what BODC did)
    2. Still waiting for model data from BODC (19/Apr/06)
  4. [Definite] Define local security roles

BODC

  1. [Definite] Install database package, including OAI
    1. Installed tomcat version of exist (19/Apr/06)
  2. [Definite] Populate MOLES instances - complete, 200 MOLES records removed and replaced with 45 with complete deployments, people+organisation records also generated
  3. [Discuss] MOLES modifications
    1. Removed all the attributes from the <dgMetadata> element in all the Moles records, to run against Kev’s new XQUERY
  4. [Definite] Populate a couple of CSML Instances
    1. In progress, working on a manually mapped point series feature
  5. [Definite] Define local security roles

BADC

  1. [Definite] Install database package, including OAI
  2. [Definite] Populate MOLES instances
    1. Complete, anything further is waiting on new BADC catalogue based on MOLES
  3. [Discuss] MOLES modifications
    1. Some issues raised with Kev as new BADC catalogue based on MOLES is implemented
  4. [Definite] Populate a couple of CSML Instances
    1. Some populated, but not sure if they correspond to any existing BADC datasets. Finding out from DL
  5. [Definite] Define local security roles

Grid-V02

Due by/Milestone: Implementation 2/Feedback 2, May 2006

PML

  1. [Definite] Get new webserver actually working
    1. Substantial job due to legacy code base
  2. [Definite] Local UserDB needed, coupled to roles
  3. [Definite] Establish which datasets are to be available and when
  4. [Definite] MOLES catalogue for instances goes live (with test user)
  5. [Definite] OAI scripts running from MOLES to OAI harvester
    1. Complete, but stability issues with OAI provider
  6. [Discuss] Security: facilitate Phil installing and testing security system at PML.
  7. [Discuss] Data Provider Requirements:
    1. communicate issues with WG team, e.g. lack of documentation.
  8. [Definite] Integrate NDG authentication with RSDAS authentication system for data access.
  9. [Definite] Populate a couple of CSML Instances.
    1. Not done yet – fixing MOLES first
  10. [Discuss] CSML issues

NOCS

  1. [Definite] Install initial security system at NOCS (with Phil)
  2. [Probable] Connect local userDB to NDG attribute authority
    1. Discussing and trialling solutions with Phil (Mar 06)
    2. Security client software working, now trying connection with NOCS database (19/Apr/06)
  3. [Definite] MOLES database goes live
  4. [Definite] OAI scripts running from MOLES to OAI harvester
  5. [Definite] Populate a couple of CSML Instances
    1. Waiting on OCCAM data (Mar 06)
    2. Still waiting (19/Apr/06)
  6. [Discuss] CSML issues
  7. [Probable] Consider caching requirements

BODC

  1. [Definite] Instantiate NDG attribute authority
    1. Instantiated, needs testing
  2. [Definite] Populate MOLES instances (which ones?)
    1. Regenerate Moles with Deployment records (including security information)
    2. Automate the BODC Moles generation software. (already around 45 datasets with deployments)
  3. [Discuss] Populate CSML instances (which ones?)
    1. Handcode 1 sea-level (points), 1 CTD (profile) instance with complete MOLES and CSML
  4. [Discuss] CSML issues

BADC

  1. [Definite] Instantiate NDG attribute authority
    1. Prototype installed and tested, working from bodc-to-badc, but problems badc-to-bodc. Full ndg aa probably not possible for may, more like alpha. (Mar 06)
    2. Works internally, waiting on solution to cross-domain cookies before can be tested externally (ticket:41) (19/Apr/06)
  2. [Definite] Populate MOLES instances (which ones?)
    1. Dependent on new badc catalogue based on moles being set up and populated. Could populate small number of specific datasets by hand if required for corresponding to csml instances. (Mar 06)
    2. Populated, but not live. Could start writing MOLES extraction (resources) (19/Apr/06)
  3. [Discuss] Populate CSML instances (which ones?)
    1. Decided list of datasets to-do. DL working on scanner which should enable more automation, at least with netcdf based data.
    2. Some populated, need to tie-in with MOLES instances (may not match up currently) (19/Apr/06)
  4. [Discuss] CSML issues
    1. Finding out from DL (Mar 06)
    2. Meeting with Dom booked (19/Apr/06)

Grid-V03

Due by/Milestone: NDG Alpha (June 2006)

Various / All groups

  1. [Definite] Discovery services working
    1. Created list of minimum things to-do to make it ‘production’. That is, good enough to take over from NERC Metadata Gateway.
    2. Done, but bounced back (milestone)
  2. [Definite] Sites providing A and B data, listed and working on portal
  3. [Definite] OAI harvesting automated at NDG portal
    1. About 75% complete (Mar 06)
    2. 90% complete (19/Apr/06)
  4. [Definite] Security infrastructure in place, but not necessarily in use
  5. [Definite] Demonstrate a successful role mapping (get an attribute certificate)
    1. BODC has most of the resources supplied and available to run the NDG Security API (March 06)
    2. Cross domain cookies issue again
  6. [Discuss] Demonstrate secured application (DX? e.g. non-BADC user able to use the DX in some way)
  7. [Definite] Feedback to Data Provider package regarding installation instructions, etc
  8. [Discuss] Establish conformance/testing procedures for releases

Grid-V04

Due by/Milestone: External Product Definition (originally August 2006, this milestone is end of Oct, may want to add a new milestone)

Various / All groups

  1. [Discuss] Security infrastructure operational for most services
  2. [Discuss] Test and review conformance/testing process for releases
  3. [Discuss] Feedback to documentation team
  4. [Discuss] Quantify available datasets and issues with further population (mini data checkpoint)
  5. [Discuss] Test installs of (secured) Data Delivery service?
  6. [Discuss] Secure MOLES browsing working

Notes for Mike (temporary while planning/updating TiD):

Not sure about these: Logging? GS at DPs? wiki:T11_ClientPackage - this looks like a final item (beta+) - maybe needs splitting up more browse (no real package - moles? discovery?) vocab (not sure which level to integrate this)

PostAlpha?? Security Integration

  • dps
  • apps (DX, Browse, GeoSPlAT?)

PreBeta? DX backend (web-services) deployment to DPs (cf. ticket:82) Security integration (delivery) CSML 2/3 CSML IO layer: PML/NOCS/BODC CSML population (inc security) (wiki:CSML_Population) Discovery portal and harvesting operational

BETA DX testing (ticket:249) DX web frontend GS web frontend (ticket:93) Security Deployment (Documentation) Security: Role-mapping between DPs complete CSML services (opendap, W*S)? MOLES population wiki:CSML_Population Discovery: web services DP: python install preparatory for packaging (ticket:300)

Post-Beta? Discovery: user testing Documentation plan? (depends on ticket:290) Security system test plan & testing General web services testing plan DP: packaging up a release

Production MOLES complete CSML complete Security fully operational (system in place for updates and new roles?) CSML services, DX, GeoSPlAT, delivery working Documentation available (promotional, DP, user, developer) Future planning (maintenance, etc) Discovery and harvesting portal operational