wiki:OWSSoftwareCollaborations

Version 2 (modified by astephen, 13 years ago) (diff)

--

Software Development Projects Involving OGC-compliant Services

Introduction

We, the climate and met scientific community (in the UK) are developing a number of software tools that will seek to expose our geospatial datasets in a manner compliant with the OGC Web Service (OWS) specifications:

  1. WCS – Web Coverage Service [Ref]
  2. WFS – Web Feature Service [Ref]
  3. WMS – Web Map Service [Ref]

We are also developing web-client applications that provide a simple user-interface:

  1. Web-clients – that can talk to OWSs

It is also worth mentioning other significant code-bases that exist:

  1. Input/Output? back-ends/APIs – often potentially running behind an OWS, or talking to it.

Purpose

The purpose of this page is to expose the software that we are building. There are three main aims that we hope to achieve:

  1. Collaborative Code Development (big savings!)
  2. Interoperability of Interfaces
  3. Identification of “holes” in the OGC specs that need to be filled to ensure they are fit for our purpose.

This page does not present a Road Map because there are different drivers and requirements for different collaborators.

Organisations Involved

The collaborations that we hope will come out of this process should include the BADC, NDG, ESSC (and possibly, the Met Office). Anyone else reading this is welcome to get in contact (mail: a.stephens AT rl.ac.uk ).

Kick-off Meeting: 01/11/06 Notes from the Kick-off meeting that started these activities are provided in rough form at [Ref: /OWSSoftwareCollaborations/KickOffMeeting] – APPENDED TO THIS DOC FOR NOW

Existing Code Bases

The following table outlines the existing developments that we are aware of, have used, tried, and are actively developing. There is a column marking whether each product already does, or could potentially, expose an OWS implementation, OWS client or useful I/O layer.

Who? What? Status? WCS? WFS? WMS? Web-client? I/O-Layer? Language? Comment. GML Application Schemas: CSML (1.0/2.0), O&M, MO App Schemas. WMS: GeoServer?, Ag’s hack (CDMS/VCS), WCS: NDG-WCS, GADS-WCS/GeoServer, WFS: GADS-GeoServer?-WCS,

LAS GeoServer? ESSC ncWMS Godiva2 DX-Server DX-Client GS-Server GS-Client

  1. DX and GS clients: Ag intro
  2. ReSC GoogleMaps/OpenLayers? map portal: currently ad-hoc, moving to OpenLayers?. Making back-end a WMS then front end could talk to any WMS. Plan to distribute back-end to other people, via NetCDF.

BL: How will it differ from THREDDS? AW: THREDDS is only WCS, not WMS. JB: Keep simple and no security etc. OpenLayers? is WCS and WFS aware. AW: WFS client is normally not aware of more than one feature. Maybe it is aware of only SimpleFeaturesProfile? – doing point, line, polygon only.

  1. GeoServer? [General chat includes WCS and WFS and WMS] – heavily set up around 2D. Source is a set of features. Can render features into a map (uses simple profile?) – based on rendering features on the land surface. But GeoServer? is now being re-engineered to support 4D data (relevant to WCS) and to support NetCDF inputs. Support WMS 1.1.1.

GeoServer? issues: JT: GeoServer? doesn’t currently connect the Feature and Coverage support. When they re-factor they’ll put it all into supporting the general feature model. And once GeoTools? has been re-factored to support the (4D) General Feature Model it should provide a harmonized view. JB: You can plug in your own modular WCS or other into the GeoServer? framework. AW: Neil will be working on getting CSML into GeoServer?. Should we plug in like JB (DEWS) or go for strategic route in core of GeoServer? (or will it not be ready in time). NB has about 6 months to work on this. Branches that we know of are: • WCS branch • Complex Feature branch (Rob Atkinson) – hacky and will die when re-factoring of GFM • Trunk will be re-factored but probably unlikely to be ready within next year

  1. MapServer?: Faster than GeoServer? (C-based not Java), doesn’t try and be clever about Feature Model. Just creates pictures. Can compile with NetCDF libs. Slow to generate maps when high-res (GDAL issue). Hence JB developed alternative. DARC have used MapServer? in their Visualisation Observatory?? GDAL is 2D limited.
  2. ESSC Java-NetCDF-WMS (NcWMS): began as python on top of cdat and apache etc. Now Java NetCDF libraries worth using (instead of GADS). Hence merged to a python implementation with:

• python to talk to CDMS • jython to talk to Java Web Application I.e. one set of code with two different back-ends.

  1. TPAC are implementing a WMS layer over LAS (new Java version).
  1. TPAC/NDG-WCS: 1.0 compliant, began as WCS wrapper over OpenDAP. Now works on CSML GridSeries? Features and can serve NetCDF as well.
  2. EESC GADS-WCS (for DEWS): Building into GeoServer?, but using GADS libraries. 1.0 compliant. Would like to get to 1.1. Asynchronous delivery is added (from WPS spec). Integrating with NDG/DEWS Security. Talking HTTP rather than SOAP.
  3. DX server: Ag talked about interface issues (SOAP, WSDL etc).

AW: Very few people have dealt with WCS in other stuff than 2D-gridded data. In theory we could use either WCS or WFS for CSML.

  1. GeoServer? at the MO: Simple GML Apps supporting only. Will be using it for DEWS – but might need to simplify DEWS feature model.

Working Together?

The following pages detail overlap where we intend to start converging our code development.

How to Co-develop?

One key issue in co-developing is what strategy we adopt. Some of the key issues are:

  • common code repositories
  • common wiki and documentation
  • merging
  • planning
  • co-ordinating tasks
  • software to do these (e.g. CVS, SVN)

Frameworks To Consider

Bryan has written some useful notes on WSGI (Web Server Gateway Interface) [REF], a python framework for standardising how web-based interactions should take place. It includes use server/client request/response, session management, threading etc etc* Appendix 1: to put in new wiki

Agenda for Our Services and OGC Web Services Meeting Version 0.4 – Ag Stephens

Attendees: A.Woolf (NDG), D. Lowe (NDG), J. Blower (ESSC), J.Tandy (MO), B.Lawrence (NDG), A.Stephens (NDG), N.Bennett (Daresbury), S.Pascoe (NDG). Time: 1000-1300, Wednesday 1st November 2006. Location: EOASD Room, , RAL. Contact for Visitors: Dominic Lowe - 01235 44(5139)

Agenda

1000-1200: What have we got and what do we know about? Discussion with possibly some quick demos if people want to. The aim of this session will be to draw up (literally) a diagram of all the components that we currently, or plan to, deploy and the external projects that we hope to borrow from. I think the cards on the table are:

Geo-clients: DX and GS clients, ReSC GoogleMaps/OpenLayers? map portal. GML Application Schemas: CSML (1.0/2.0), O&M, MO App Schemas. WMS: GeoServer?, ReSC CDMS-WMS, Ag’s hack (CDMS/VCS), ESSC Java-NetCDF-WMS (NcWMS)? WCS: NDG-WCS, GADS-WCS/GeoServer, DX server, version changes. WFS: GADS-WCS/GeoServer, DX server, version changes. Filter Encoding (applied to other services) WPS: ? Martin Juckes proposal, JB has looked at spec, MO use, DEWS use? Projects: DEWS, MO projects, GALEON, SEEGRID, MOTIIVE, CCLRC-ESC, TPAC, GALEON) ShapeChange? issues: CSML-compatibility? auto-generation of parsing/demarshalling code? integration into EA? GML 3.2-compliance?

Given that the conversation will no doubt jump between these because of the crossover I don’t think we need to worry about treating them as sequential agenda items.

Things that we also need to cover:

  • queuing systems
  • parallel systems
  • animations rather than images

1200-1300: Where do we think we are going? There are clearly a lot of possible futures. However, we now have enough experience to be able to guess the future. The aim of this session is to look at how we can achieve convergence in the mid to long term. Which components do we imagine will stand the test of time?

Ag and Bryan to exit at 1300 but others welcome to carry on.

Table on current Versions of OGC Specs (is this right?) Spec Commonly implemented Current Draft WFS 1.0 1.1.0 (May 2005) + Corrigendum (Oct 2005) 1.2 (not seen) DAQM and filter encoding 1.2 WCS - 1.0 (August 2003) + Corrigendum (Oct 2005) 1.1 (voting on it now) WMS 1.1.1 1.3.0 (March 2006) + App Profile for EO Products (July 2006) ISO19128 standardised ? WPS 0.4.0 (Sep 2005) - DRAFT ? Filter Encoding 1.1.0 (May 2005) ?

Geo-clients:

  1. DX and GS clients: Ag intro
  2. ReSC GoogleMaps/OpenLayers? map portal: currently ad-hoc, moving to OpenLayers?. Making back-end a WMS then front end could talk to any WMS. Plan to distribute back-end to other people, via NetCDF.

BL: How will it differ from THREDDS? AW: THREDDS is only WCS, not WMS. JB: Keep simple and no security etc. OpenLayers? is WCS and WFS aware. AW: WFS client is normally not aware of more than one feature. Maybe it is aware of only SimpleFeaturesProfile? – doing point, line, polygon only.

GML Application Schemas:

  1. CSML (1.0/2.0): GML App Schema to define FTs for climate science data. Re-factoring to V2.0 which increases number of FTs, becomes more compatible with O&M model, defines some “operations” associated with FTs.
  2. O&M: Going to be a new OGC Spec. It is an App Schema developed by OGC. It uses standard patterns for recording information from measurements and simulations etc.
  3. MO App Schemas: Need to expose meteorological information. But limited to Simple

Services: SOS: Will allow you to expose O&M features.

WMS Spec: The WMS Spec: 1.3.0 is latest. The major changes from 1.1.0 are: • data is a set of maps, supports time and elevation and other arbitrary dimensions. (i.e. XY x any number of other dimensions). • supports a number of CRSs. • Changed code for some projections and then swapped lat/lon. • a layer is now 4D – i.e. is a variable (or parameter or field) and these can be nested in layers that are essentially datasets. Very few implementations are truly compliant with the spec. Mpeg (i.e. animation) is valid output format. Possible holes in spec are: • cannot (easily) match a range to a colourmap • capabilities document is monolithic: o layer definitions allow nesting but could expose large XML unnecessarily. o But GetCapabilities? has a timestamp so the client can cache the returned XML and then only get new version if it has changed.

WMS Implementations:

  1. GeoServer? [General chat includes WCS and WFS and WMS] – heavily set up around 2D. Source is a set of features. Can render features into a map (uses simple profile?) – based on rendering features on the land surface. But GeoServer? is now being re-engineered to support 4D data (relevant to WCS) and to support NetCDF inputs. Support WMS 1.1.1.

GeoServer? issues: JT: GeoServer? doesn’t currently connect the Feature and Coverage support. When they re-factor they’ll put it all into supporting the general feature model. And once GeoTools? has been re-factored to support the (4D) General Feature Model it should provide a harmonized view. JB: You can plug in your own modular WCS or other into the GeoServer? framework. AW: Neil will be working on getting CSML into GeoServer?. Should we plug in like JB (DEWS) or go for strategic route in core of GeoServer? (or will it not be ready in time). NB has about 6 months to work on this. Branches that we know of are: • WCS branch • Complex Feature branch (Rob Atkinson) – hacky and will die when re-factoring of GFM • Trunk will be re-factored but probably unlikely to be ready within next year

  1. MapServer?: Faster than GeoServer? (C-based not Java), doesn’t try and be clever about Feature Model. Just creates pictures. Can compile with NetCDF libs. Slow to generate maps when high-res (GDAL issue). Hence JB developed alternative. DARC have used MapServer? in their Visualisation Observatory?? GDAL is 2D limited.
  2. ESSC Java-NetCDF-WMS (NcWMS): began as python on top of cdat and apache etc. Now Java NetCDF libraries worth using (instead of GADS). Hence merged to a python implementation with:

• python to talk to CDMS • jython to talk to Java Web Application I.e. one set of code with two different back-ends.

  1. TPAC are implementing a WMS layer over LAS (new Java version).

WCS Spec: Voting on it now. One problem is that implementations tend to be 2D focussed – until latest version gets implemented. DL gave overview of changes to 1.1. Issues about CRS and dimensionality selection.

WCS Implementations:

  1. TPAC/NDG-WCS: 1.0 compliant, began as WCS wrapper over OpenDAP. Now works on CSML GridSeries? Features and can serve NetCDF as well.
  2. EESC GADS-WCS (for DEWS): Building into GeoServer?, but using GADS libraries. 1.0 compliant. Would like to get to 1.1. Asynchronous delivery is added (from WPS spec). Integrating with NDG/DEWS Security. Talking HTTP rather than SOAP.
  3. DX server: Ag talked about interface issues (SOAP, WSDL etc).

AW: Very few people have dealt with WCS in other stuff than 2D-gridded data. In theory we could use either WCS or WFS for CSML.

WFS Spec: Could WFS spec use getGmlObject to return a binary file (NetCDF)? We should look into this.

WFS Implementations:

  1. DX server
  2. GeoServer? at the MO: Simple GML Apps supporting only. Will be using it for DEWS – but might need to simplify DEWS feature model.

Filter Encoding (applied to other services) WPS: ? Martin Juckes proposal, JB has looked at spec, MO use, DEWS use? Projects: DEWS, MO projects, GALEON: Moving into a phase looking at further integration tests etc, hooking into NcML and CSML. We will contribute where we can. THREDDS: Has a WCS interface. SEEGRID, MOTIIVE, RISE, CCLRC-ESC, TPAC

ShapeChange? issues: ShapeChange? allows the generation of a GML App Schema from a UML model based on the rules set in ISO19xxx. You can export UML to XMI, import to ShapeChange? and it should be able to create an App Schema. However, it is limited to simple features. Main point is that you can map onto existing schemas so that you can re-use elements from other ISO standards and/or other GML schemas etc. Mappings to things like CSML are missing. GML 3.2-mappings not available until re-factored, and someone creates them. No specific plans for integration into EA. XMI 2 is going to be a clear standard. Planning to use the Eclipse MOF model.

Outputs we want from this meeting:

  1. Whiteboard roadmap
  2. Notes (Jon, Ag, Jeremy, Bryan?)
  3. Key issue: interoperability tests – interfaces that can talk to eachother.

ACTIONS: DL: Raise issue of asynchronous results. Need to look at WPS version (as used in DEWS by JB).

Whiteboard roadmap

  1. Notes (Jon, Ag, Jeremy, Bryan?)
  1. Key issue: interoperability tests – interfaces that can talk to eachother.

ACTIONS:

DL: Raise issue of asynchronous results. Need to look at WPS version (as used in DEWS by JB).