wiki:NERCPortals

Version 25 (modified by mkochan, 11 years ago) (diff)

--

The NERC Portals project

Since it's inception the NERC Portals project has started to be called the Data Portals Prototype Project (DPPP) across the participating institutions.

See

Conversion from CSML to KML

User manual

Since CSML currently lacks the constructs for describing the context for the CSML features, the actual mechanisms used to convert data from CSML to KML do use limited amount of CSML as a data source. The rest comes from different languages derived from GML. Hence this project became rather a demonstration of how OGC web services can be integrated into a common framework, which has a potential of working using purely CSML data, once CSML matures enough to contain all information in it.

The functionality is split into two eggs:

  • csml2kml - provides static functionality, with the output being KML stored in files (*.kml, *.kmz),
  • csml2kmlpylon - is a web server (to be run by the BADC) providing web services which brings dynamic content to the viewing experience in Google Earth via HTTP.

Installation

Both eggs require the pylab and cElementTree modules.

The installation procedure is:

  1. Use the standard easy_install command to install the eggs. csml2kml and csml2kmlpylon do not have to be installed on the same machine.
  2. Before starting the csml2kmlpylon server, edit the config file (see csml2kmlpylon section concerning the config file).

Usage of the csml2kml scripts

The wms2kml.py script

Run this script as:

python wms2kml.py filename.conf.xml

where filename.conf.xml stands for an XML config file.

The script interrogates a WMS web service using the GetCapabilities call and thus acquires a <wms:Capabilities> XML document from it, which contains description of WMS layers served by the WMS web service. Then, it exports this into a hierarchy of embedded directories forming a structure which contains a hierarchy of expandable KMZ documents linked together using the <kml:NetworkLink> tag, and loaded on demand. The files can then be served to the outer world via HTTP.

An example format of the config file is as follows (example config files can be found in the egg's csml2kml/config sub-directory):

<WMSLayers2KML>
  <WMSRequest>
    <URL>http://www-devel.ipcc-data.org/maps/wms/obs</URL>
    <ServiceVersion>1.1.1</ServiceVersion>
    <ImageFormat>image/png</ImageFormat>
    <ImageWidth>960</ImageWidth>
    <ImageHeight>480</ImageHeight>
    <CRS>EPSG:4326</CRS>
  </WMSRequest>
  <OutputRootDirectory>/home/users/mkochan/svn-root/kml/csml2kml/output/wms-obs2</OutputRootDirectory>
  <ServerRootDirectory>http://bond.badc.rl.ac.uk:8089</ServerRootDirectory>
</WMSLayers2KML>

The field URL is a base URL of the WMS web service serving the intended dataset.

The stations2kml.py script

Run this script as:

python stations2kml.py filename.conf.xml

where filename.conf.xml stands for an XML config file.

This script interrogates a WFS to get a list of (meteorological) stations, recorded using an element <np:Station> defined for the purposes of this project. It generates a KMZ file that contains a number of <kml:Placemark> elements, each standing for a single station. Each station has has a balloon filled according to a defined template, which refers to a URL of the running csml2kmlpylon pylon. Thus dynamic content about each station can be presented.

An example config file looks as follows:

<Stations2KML>
  <DocumentName>MIDAS stations (first 100 of them)</DocumentName>
  <UseRegions>yes</UseRegions>
  <GetAllStationsRequestURL><![CDATA[http://130.246.76.98:8084/geoserver/wfs?request=getFeature&service=wfs&version=1.1.0&typ
ename=np:Station&maxFeatures=100]]></GetAllStationsRequestURL>
  <BalloonTemplate>&lt;h2&gt;Station $[station_name]&lt;/h2&gt;Click here to see the list of &lt;a href=&quot;http://bond.badc.rl.ac.uk:8089/csmlGrapher/list?station_name=$[station_name]&quot;&gt;CSML features associated with this station&lt;/a&gt; (this will open a window of your default web browser).</BalloonTemplate>
  <StationData>
    <Datum name="station_id">#ID#</Datum>
    <Datum name="station_name">#NAME#</Datum>
  </StationData>
   <OutputKmzFilename>/home/users/mkochan/svn-root/kml/csml2kml/output/exampleps_stations.4.kmz</OutputKmzFilename>
</Stations2KML>

Details for some non-obvious settings are as follows:

  • UseRegions - determines whether to use embedding into regions.
    • If set to no, stations will be recorded simply as a list of KML placemarks, and will be visible from all altitudes. May cause slow rendering for large number of placemarks.
    • If set to yes, stations will be divided by recursively splitting the the covered are into 4 smaller KML regions. The actual station KML placemarks will be placed at the bottom of the hierarchy. Therefore, they will be forced to be visible only for closer zooms onto the ground, so rendering will be quick.
  • GetAllStationsRequestURL - is an URL at which the <np:Station> list can be retrieved
  • BalloonTemplate - is a template that is used for the balloon of every station/placemark. It should contain an URL (as part of an HTML anchor) linking to the csml2kmlpylon Web service.
  • StationData - contains station-specific data that are substituted in the balloon template. The #ID# and #NAME# fields get replaced by the script with station identifier (such as "midas.station.2500") and name (such as "FAIR ISLE"), respectively. Note that a station's latitude and longitute need not be shown in the balloon, as they can be found by right-clicking on the station placemark.

Usage of the csml2kmlpylon web service

The pylon currently contains only a single controller called csmlGrapher.py.

After being installed, the server can be run by the command:

paste serve $PATH_TO_CSML2KMLPYLON/development.ini

The controller csmlGrapher.py also has a config file which needs to be specified in development.ini in the [app:main] section e.g.:

...
[app:main]
...
csmlGrapher.configfile = %(here)s/csml2kmlpylon/config/csml2kmlpylon.conf.xml
...

This also shows the preferred location of the config file.

Format of the config file is as follows:

<GrapherWebService>
  <DisplayIntervalStart>2006-6-1</DisplayIntervalStart>
  <DisplayIntervalEnd>2006-7-1</DisplayIntervalEnd>
  <ServedFromURL>http://superglue.badc.rl.ac.uk:8084/csmlGrapher</ServedFromURL>
  <Dataset id="midas" name="MIDAS">
    <GeoServerURL>http://bond.badc.rl.ac.uk:8084/geoserver/wfs</GeoServerURL>
    <DatetimeQueryFormat>1-JAN-1970</DatetimeQueryFormat>
  </Dataset>
  <Dataset id="ecn" name="ECN">
    <GeoServerURL>http://dppp.nerc-lancaster.ac.uk:8080/geoserver/wfs</GeoServerURL>
    <DatetimeQueryFormat>UTC_TIMESTAMP</DatetimeQueryFormat>
  </Dataset>
</GrapherWebService>

The fiels have the following meanings: The interval of dates determines which range of dates to request from the GeoServer? and to plot. The GeoServerURL is the URL to be used as a base of the GetFeatureInfo? call.

Assuming that the pylon is served at URL http://server:port, the web service can be used in two ways:

  • Using the list sub-service to provide a listing of CSML features available for a given station:
    http://server:port/csmlGrapher/list?station_name=STATION%20NAME
    
    Note that the station is identified by its name (e.g. "FAIR ISLE", rather than station ID (e.g. "midas.station.3"); this is due to how GeoServer? identifies the stations.
  • Using the plot sub-service to plot a specific CSML feature of a specific station:
    http://server:port/csmlGrapher/list?station_name=STATION%20NAME&feature_id=feature_id
    
    The HTML produced by csmlGrapher's list sub-service produces links containing the corresponding plot sub-service URLs.

Development notes

It is recommended that for future/ongoing development purposes, a "development egg" is created, by running:

python setup.py develop --install-dir <install_dir_name>

using the setup.py files in the root directories of the csml2kml and csml2kmlpylon packages. The egg-links will get created in <install_dir_name> directory; the $PYTHONPATH environment variable will have to contain this directory.

Development notes for csml2kml

The egg's csml2kml directory contains all Python modules, and the following 4 sub-directories:

  • scripts - contains Python stand-alone scripts, runnable from command line, which both use a config file as a means of input
  • tests - contains tests, which simply print output (rather than fail/pass messages), and are runnable as python test_filename.py, but please look at the contents first to see what the test does
  • config - a preferred location for config files for the scripts, in XML format, which define locations of used Web services etc.
  • output - empty after installation, optionally usable for output of scripts and tests

The modules...

There is a naming convention that wrappers of entities which normally exist as XML elements are prefixed with XML spacename. For instance, a class WMSLayer is a wrapper for an element <wms:Layer>. Wrappers for KML elements (in kml namespace), which are used for output, contain a build() method, which builds each instance into an ElementTree.Element object. On the contrary, wrappers for elements in wms and np namespaces contain parseXML() method, as they are used for input.

Development notes for csml2kmlpylon