wiki:NERCPortals

Version 47 (modified by domlowe, 13 years ago) (diff)

fixing svn link

The NERC Portals project

Since it's inception the NERC Portals project has started to be called the Data Portals Prototype Project (DPPP) across the participating institutions.

See

Conversion from CSML to KML: User manual

Since CSML currently lacks the constructs for describing the context for the CSML features, the actual mechanisms used to convert data from CSML to KML do use limited amount of CSML as a data source. The rest comes from different languages derived from GML. Hence this project became rather a demonstration of how OGC web services can be integrated into a common framework, which has a potential of working using purely CSML data, once CSML matures enough to contain all information in it.

The functionality is split in two:

  • csml2kml (an egg) - acts as an API, but accompanied by command-line runnable scripts. Provides static CSML-to-KML conversion functionality, with the output being KML documents stored in files (*.kml, *.kmz) or directory hierarchies of files connected via KML NetworkLink's.
  • csml2kmlpylon (a Pylon application) - is a web server (to be run by the BADC) serving a csmlGrapher web service that renders dynamic HTTP content on top of static data in KML files being viewed in Google Earth.

Installation on a target machine

Both csml2kml and csml2kmlpylon require matplotlib, cElementTree and csml.

Installation of csml2kml

The procedure is:

  1. Use the standard easy_install command to install latest version of the egg on the target machine. 
    easy_install -f http://ndg.nerc.ac.uk/dist csml2kml

This will install the egg and also install two scripts (called wms2kml and stations2kml).

Temporary storage and archival of KML results

For daily use as a directory for KML output, it is recommended to use the following directory with svn:ignore='*' set on it:  http://proj.badc.rl.ac.uk/ndg/browser/DPPP/kml/kml-output

There is now a directory in the repository:  http://proj.badc.rl.ac.uk/ndg/browser/DPPP/kml/data-archive usable as space for storing important generated results as tarballs.

To re-generate the KML files you would need to do the following:

  1. Check out the below *.conf.xml files from the repository at :

 svn+ssh://proj.badc.rl.ac.uk/disks/glue1/svn-repository/ndg/DPPP/kml/python/csml2kml/csml2kml/config See further below for format details.

  1. Edit the output destination directories in them and make sure you have created the output directories.
  2. Run the following commands (as of 2 April 2008, these are the existing datasets): 
    wms2kml ar4.wms2kml.conf.xml
wms2kml obs.wms2kml.conf.xml
stations2kml midas.csml2kml.conf.xml               
stations2kml ecn.csml2kml.conf.xml

Installation of csml2kmlpylon

  1. Check out the Pylon application from the repository, at Subversion URL: 
    svn+ssh://proj.badc.rl.ac.uk/ndgsvn/DPPP/kml/python/csml2kmlpylon
  2. Edit the config file, csml2kmlpylon/config/csml2kmlpylon.conf.xml (relative to working copy root). See section  Usage of the {{{csml2kmlpylon}}} web service for details on the config file.
  3. If the KML/KMZ files pointing to this web service are not yet served from elsewhere, you can put them (or symlinks to them) in the csml2kmlpylon/public directory.
  4. Run the service by executing: 
    paster serve deploy.ini --deamon
    Alternatively, if you want to do development on the Pylon (with Pylon debug mode turned on), run the command: 
    paster serve development.ini --deamon

Usage of the csml2kml scripts

The wms2kml.py script

Run this script as: 

python wms2kml.py filename.conf.xml

where filename.conf.xml stands for an XML config file.

The script interrogates a WMS web service using the GetCapabilities call and thus acquires a <wms:Capabilities> XML document from it, which contains description of WMS layers served by the WMS web service. Then, it exports this into a hierarchy of embedded directories forming a structure which contains a hierarchy of expandable KMZ documents linked together using the <kml:NetworkLink> tag, and loaded on demand. The files can then be served to the outer world via HTTP.

An example format of the config file is as follows (example config files can be found in the egg's csml2kml/config sub-directory): 

<WMSLayers2KML>
  <WMSRequest>
    <URL>http://www-devel.ipcc-data.org/maps/wms/obs</URL>
    <ServiceVersion>1.1.1</ServiceVersion>
    <ImageFormat>image/png</ImageFormat>
    <ImageWidth>960</ImageWidth>
    <ImageHeight>480</ImageHeight>
    <CRS>EPSG:4326</CRS>
  </WMSRequest>
  <OutputRootDirectory>/home/users/mkochan/svn-root/kml/csml2kml/output/wms-obs2</OutputRootDirectory>
  <ServerRootDirectory>http://bond.badc.rl.ac.uk:8089</ServerRootDirectory>
</WMSLayers2KML>

The field URL is a base URL of the WMS web service serving the intended dataset.

The stations2kml.py script

Run this script as: 

python stations2kml.py filename.conf.xml

where filename.conf.xml stands for an XML config file.

This script interrogates a WFS to get a list of (meteorological) stations, recorded using an element <np:Station> defined for the purposes of this project. It generates a KMZ file that contains a number of <kml:Placemark> elements, each standing for a single station. Each station has has a balloon filled according to a defined template, which refers to a URL of the running csml2kmlpylon pylon. Thus dynamic content about each station can be presented.

An example config file looks as follows: 

<Stations2KML>
  <DocumentName>MIDAS stations</DocumentName>
  <UseRegions>yes</UseRegions>
  <GetAllStationsRequestURL><![CDATA[http://bond.badc.rl.ac.uk:8084/geoserver/wfs?request=getFeature&service=wfs&version=1.1.0&typename=np:Station&maxFeatures=100000]]></GetAllStationsRequestURL>
  <BalloonTemplate>&lt;h2&gt;Station $[station_name]&lt;/h2&gt;Click here to see the list of &lt;a href=&quot;http://superglue.badc.rl.ac.uk:8084/csmlGrapher/list?dataset_id=midas&amp;station_name=$[station_name]&quot;&gt;CSML features associated with this station&lt;/a&gt; (this will open a window of your default web browser).</BalloonTemplate>
  <StationData>
    <Datum name="station_id">#ID#</Datum>
    <Datum name="station_name">#NAME#</Datum>
  </StationData>
  <DisplayIntervalStart>2006-6-1</DisplayIntervalStart>
  <DisplayIntervalEnd>2006-8-1</DisplayIntervalEnd>
  <OutputKmzFilename>/home/users/mkochan/svn-root/kml/csml2kml/output/midas_stations.kmz</OutputKmzFilename>
</Stations2KML>

Details for some non-obvious settings are as follows:

  • UseRegions - determines whether to use embedding into regions.
    • If set to no, stations will be recorded simply as a list of KML placemarks, and will be visible from all altitudes. May cause slow rendering for large number of placemarks.
    • If set to yes, stations will be divided by recursively splitting the the covered are into 4 smaller KML regions. The actual station KML placemarks will be placed at the bottom of the hierarchy. Therefore, they will be forced to be visible only for closer zooms onto the ground, so rendering will be quick.
  • GetAllStationsRequestURL - is an URL at which the <np:Station> list can be retrieved
  • BalloonTemplate - is a template that is used for the balloon of every station/placemark. It should contain an URL (as part of an HTML anchor) linking to the csml2kmlpylon Web service.
  • StationData - contains station-specific data that are substituted in the balloon template. The #ID# and #NAME# fields get replaced by the script with station identifier (such as "midas.station.2500") and name (such as "FAIR ISLE"), respectively. Note that a station's latitude and longitute need not be shown in the balloon, as they can be found by right-clicking on the station placemark.
  • DisplayIntervalStart and DisplayIntervalEnd specify the time internal, which is used in marking some stations red: the stations that contain at least one CSML feature which has been measured during this time interval, will be painted red; the rest will be white.

Usage of the csml2kmlpylon web service

Assuming that the pylon is served at URL http://server:port, the web service can be used in two ways:

  • Using the list sub-service to provide a listing of CSML features available for a given station: 
    http://server:port/csmlGrapher/list?dataset_id=DATASET_ID&station_name=STATION%20NAME
    The "dataset" here refers to an URL of a WFS server from which station data can be fetched. Each station is identified by its name (e.g. "FAIR ISLE", rather than station ID (e.g. "midas.station.3"); this is due to how our GeoServer? identifies the stations.
  • Using the plot sub-service to plot a specific CSML feature of a specific station: 
    http://server:port/csmlGrapher/list?dataset_id=DATASET_ID&station_name=STATION%20NAME&feature_id=feature_id
    The HTML produced by csmlGrapher's list sub-service produces links containing the corresponding plot sub-service URLs and optionally also verbatim links to WFS that return underlying CSML if the user has access to that WFS.

Please make sure you use Firefox to view the generate web content (possibly by choosing to use an external viewer in Google Earth's Tools/Options? menu) as IE7 does not show the SQL errors properly.

The pylon currently contains only a single controller called csmlGrapher.py.

After being installed, the server can be run by the command: 

paste serve $PATH_TO_CSML2KMLPYLON/deploy.ini

The controller csmlGrapher.py also has a config file which needs to be specified in development.ini in the [app:main] section e.g.: 

...
[app:main]
...
csmlGrapher.configfile = %(here)s/csml2kmlpylon/config/csml2kmlpylon.conf.xml
...

Format of the config file is as follows: 

<GrapherWebService>
  <DisplayIntervalStart>2006-6-1</DisplayIntervalStart>
  <DisplayIntervalEnd>2006-7-1</DisplayIntervalEnd>
  <ServedFromURL>http://superglue.badc.rl.ac.uk:8084/csmlGrapher</ServedFromURL>
  <Dataset id="midas" name="MIDAS">
    <GeoServerURL>http://bond.badc.rl.ac.uk:8084/geoserver/wfs</GeoServerURL>
    <DatetimeQueryFormat>1-JAN-1970</DatetimeQueryFormat>
    <IsCSMLLinkVisible>yes</IsCSMLLinkVisible>
  </Dataset>
  <Dataset id="ecn" name="ECN">
    <GeoServerURL>http://dppp.nerc-lancaster.ac.uk:8080/geoserver/wfs</GeoServerURL>
    <DatetimeQueryFormat>UTC_TIMESTAMP</DatetimeQueryFormat>
    <IsCSMLLinkVisible>yes</IsCSMLLinkVisible>
  </Dataset>
</GrapherWebService>

The elements have the following meanings:

  • DisplayIntervalStart and DisplayIntervalEnd determine the interval for which the data will be plotted, and only those station features will be listed which have been collected during the whole span of this interval.
  • ServedFromURL is the network path to the csmlGrapher service (not just the Pylon)
  • Each Dataset element stands for one WFS acting as a source of data.
    • GeoServerURL contains the URL of the WFS/GeoServer, used as a base of the GetFeatureInfo? call.
    • DatetimeQueryFormat specifies which date format to specify date intervals in queries.
    • IsCSMLLinkVisible (use values yes or no) specifies whether to show the feature query URLs as links in the station feature listing.

Source code notes

This section describes how source code is organised.

Source code notes for csml2kml

The egg's csml2kml directory contains all Python modules, and the following 3 sub-directories:

  • scripts - contains Python stand-alone scripts, runnable from command line, which both use a config file as a means of input
  • tests - contains tests, which simply print output (rather than fail/pass messages), and are runnable as python test_filename.py, but please look at the contents first to see what the test does
  • config - a preferred location for config files for the scripts, in XML format, which define locations of used Web services etc.

The EpyDoc for the modules is available at  http://glue.badc.rl.ac.uk/doc/dppp/epydoc/csml2kml/index.html.

There is a naming convention that wrappers of entities which normally exist as XML elements are prefixed with XML spacename. For instance, a class WMSLayer is a wrapper for an element <wms:Layer>. Wrappers for KML elements (in kml namespace), which are used for output, contain a build() method, which builds each instance into an ElementTree.Element object. On the contrary, wrappers for elements in wms and np namespaces contain parseXML() method, as they are used for input.

Source code notes for csml2kmlpylon

The EpyDoc for the whole Pylon is available at  http://glue.badc.rl.ac.uk/doc/dppp/epydoc/csml2kmlpylon/index.html (especially look at controllers.csmlGrapher).