= The NERC Portals project =

Since it's inception the NERC Portals project has started to be called the Data Portals Prototype Project (DPPP) across the participating institutions.

See
 - CedaDevEnvironment being created on {{{bond}}}.

= Conversion from CSML to KML =

== User manual ==

Since CSML currently lacks the constructs for describing the context for the CSML features, the actual mechanisms used to convert data from CSML to KML do use limited amount of CSML as a data source. The rest comes from different languages derived from GML. Hence this project became rather a demonstration of how OGC web services can be integrated into a common framework, which has a potential of working using purely CSML data, once CSML matures enough to contain all information in it.

The functionality is split in two:
 - {{{csml2kml}}} (an egg) - acts as an API, but accompanied by command-line runnable scripts. Provides static CSML-to-KML conversion functionality, with the output being KML documents stored in files (*.kml, *.kmz) or directory hierarchies of files connected via KML NetworkLink's.
 - {{{csml2kmlpylon}}} (a Pylon application) - is a web server (to be run by the BADC) serving a {{{csmlGrapher}}} web service that renders dynamic HTTP content on top of static data in KML files being viewed in Google Earth.

=== Installation on a target machine ===

Both {{{csml2kml}}} and {{{csml2kmlpylon}}} require the {{{pylab}}} and {{{cElementTree}}} modules, and {{{csml}}}.

==== Installation of {{{csml2kml}}} ====

Use the standard {{{easy_install}}} command to install latest version of the egg on the target machine.

==== Installation of {{{csml2kmlpylon}}} ====

  1. Check out the Pylon application from the repository, at Subversion URL:
{{{
svn+ssh://proj.badc.rl.ac.uk/disks/glue1/svn-repository/ndg/DPPP/kml/csml2kml/python/csml2kmlpylon
}}}
  2. Edit the config file, {{{csml2kmlpylon/config/csml2kmlpylon.conf.xml}}} (relative to working copy root)
  3. Run the service by executing:
{{{
paster serve deploy.ini --deamon
}}}
     Alternatively, if you want to do development on the Pylon (with Pylon debug mode turned on), run the command:
{{{
paster serve development.ini --deamon
}}}

=== Usage of the {{{csml2kml}}} scripts ===

==== The {{{wms2kml.py}}} script ====

Run this script as:
{{{
python wms2kml.py filename.conf.xml
}}}
where {{{filename.conf.xml}}} stands for an XML config file.

The script interrogates a WMS web service using the {{{GetCapabilities}}} call and thus acquires a <wms:Capabilities> XML document from it, which contains description of WMS layers served by the WMS web service. Then, it exports this into a hierarchy of embedded directories forming a structure which contains a hierarchy of expandable KMZ documents linked together using the ''<kml:NetworkLink>'' tag, and loaded on demand. The files can then be served to the outer world via HTTP.

An example format of the config file is as follows (example config files can be found in the egg's {{{csml2kml/config}}} sub-directory):
{{{
<WMSLayers2KML>
  <WMSRequest>
    <URL>http://www-devel.ipcc-data.org/maps/wms/obs</URL>
    <ServiceVersion>1.1.1</ServiceVersion>
    <ImageFormat>image/png</ImageFormat>
    <ImageWidth>960</ImageWidth>
    <ImageHeight>480</ImageHeight>
    <CRS>EPSG:4326</CRS>
  </WMSRequest>
  <OutputRootDirectory>/home/users/mkochan/svn-root/kml/csml2kml/output/wms-obs2</OutputRootDirectory>
  <ServerRootDirectory>http://bond.badc.rl.ac.uk:8089</ServerRootDirectory>
</WMSLayers2KML>
}}}
The field {{{URL}}} is a base URL of the WMS web service serving the intended dataset.

==== The {{{stations2kml.py}}} script ====

Run this script as:
{{{
python stations2kml.py filename.conf.xml
}}}
where {{{filename.conf.xml}}} stands for an XML config file.

This script interrogates a WFS to get a list of (meteorological) stations, recorded using an element ''<np:Station>'' defined for the purposes of this project. It generates a KMZ file that contains a number of ''<kml:Placemark>'' elements, each standing for a single station. Each station has has a balloon filled according to a defined template, which refers to a URL of the running {{{csml2kmlpylon}}} pylon. Thus dynamic content about each station can be presented.

An example config file looks as follows:
{{{
<Stations2KML>
  <DocumentName>MIDAS stations</DocumentName>
  <UseRegions>yes</UseRegions>
  <GetAllStationsRequestURL><![CDATA[http://bond.badc.rl.ac.uk:8084/geoserver/wfs?request=getFeature&service=wfs&version=1.1.0&typename=np:Station&maxFeatures=100000]]></GetAllStationsRequestURL>
  <BalloonTemplate>&lt;h2&gt;Station $[station_name]&lt;/h2&gt;Click here to see the list of &lt;a href=&quot;http://superglue.badc.rl.ac.uk:8084/csmlGrapher/list?dataset_id=midas&amp;station_name=$[station_name]&quot;&gt;CSML features associated with this station&lt;/a&gt; (this will open a window of your default web browser).</BalloonTemplate>
  <StationData>
    <Datum name="station_id">#ID#</Datum>
    <Datum name="station_name">#NAME#</Datum>
  </StationData>
  <DisplayIntervalStart>2006-6-1</DisplayIntervalStart>
  <DisplayIntervalEnd>2006-8-1</DisplayIntervalEnd>
  <OutputKmzFilename>/home/users/mkochan/svn-root/kml/csml2kml/output/midas_stations.kmz</OutputKmzFilename>
</Stations2KML>
}}}
Details for some non-obvious settings are as follows:
 - {{{UseRegions}}} - determines whether to use embedding into regions.
   * If set to {{{no}}}, stations will be recorded simply as a list of KML placemarks, and will be visible from all altitudes. May cause slow rendering for large number of placemarks.
   * If set to {{{yes}}}, stations will be divided by recursively splitting the the covered are into 4 smaller KML regions. The actual station KML placemarks will be placed at the bottom of the hierarchy. Therefore, they will be forced to be visible only for closer zooms onto the ground, so rendering will be quick.
  - {{{GetAllStationsRequestURL}}} - is an URL at which the ''<np:Station>'' list can be retrieved
  - {{{BalloonTemplate}}} - is a template that is used for the balloon of every station/placemark. It should contain an URL (as part of an HTML anchor) linking to the {{{csml2kmlpylon}}} Web service.
  - {{{StationData}}} - contains station-specific data that are substituted in the balloon template.  The {{{#ID#}}} and {{{#NAME#}}} fields get replaced by the script with station identifier (such as "{{{midas.station.2500}}}") and name (such as "{{{FAIR ISLE}}}"), respectively. Note that a station's latitude and longitute need not be shown in the balloon, as they can be found by right-clicking on the station placemark.
  - {{{DisplayIntervalStart}}} and {{{DisplayIntervalEnd}}} specify the time internal, which is used in marking some stations red: the stations that contain ''at least one'' CSML feature which has been measured ''during this time interval'', will be painted red; the rest will be white.
=== Usage of the {{{csml2kmlpylon}}} web service ===

Assuming that the pylon is served at URL ''{{{http://server:port}}}'', the web service can be used in two ways:
 - Using the {{{list}}} sub-service to provide a listing of CSML features available for a given station:
{{{
http://server:port/csmlGrapher/list?dataset_id=DATASET_ID&station_name=STATION%20NAME
}}}
   The "dataset" here refers to an URL of a WFS server from which station data can be fetched. Each station is identified by its name (e.g. "{{{FAIR ISLE}}}", rather than station ID (e.g. "{{{midas.station.3}}}"); this is due to how our GeoServer identifies the stations.
 - Using the {{{plot}}} sub-service to plot a specific CSML feature of a specific station:
{{{
http://server:port/csmlGrapher/list?dataset_id=DATASET_ID&station_name=STATION%20NAME&feature_id=feature_id
}}}
  The HTML produced by csmlGrapher's {{{list}}} sub-service produces links containing the corresponding {{{plot}}} sub-service URLs and optionally also verbatim links to WFS that return underlying CSML if the user has access to that WFS.

Please make sure you use Firefox to view the generate web content (possibly by choosing to use an external viewer in Google Earth's ''Tools/Options'' menu) as IE7 does not show the SQL errors properly.

The pylon currently contains only a single controller called {{{csmlGrapher.py}}}.

After being installed, the server can be run by the command:
{{{
paste serve $PATH_TO_CSML2KMLPYLON/deploy.ini
}}}
The controller {{{csmlGrapher.py}}} also has a config file which needs to be specified in {{{development.ini}}} in the {{{[app:main]}}} section e.g.: 
{{{
...
[app:main]
...
csmlGrapher.configfile = %(here)s/csml2kmlpylon/config/csml2kmlpylon.conf.xml
...
}}}

Format of the config file is as follows:
{{{
<GrapherWebService>
  <DisplayIntervalStart>2006-6-1</DisplayIntervalStart>
  <DisplayIntervalEnd>2006-7-1</DisplayIntervalEnd>
  <ServedFromURL>http://superglue.badc.rl.ac.uk:8084/csmlGrapher</ServedFromURL>
  <Dataset id="midas" name="MIDAS">
    <GeoServerURL>http://bond.badc.rl.ac.uk:8084/geoserver/wfs</GeoServerURL>
    <DatetimeQueryFormat>1-JAN-1970</DatetimeQueryFormat>
    <IsCSMLLinkVisible>yes</IsCSMLLinkVisible>
  </Dataset>
  <Dataset id="ecn" name="ECN">
    <GeoServerURL>http://dppp.nerc-lancaster.ac.uk:8080/geoserver/wfs</GeoServerURL>
    <DatetimeQueryFormat>UTC_TIMESTAMP</DatetimeQueryFormat>
    <IsCSMLLinkVisible>yes</IsCSMLLinkVisible>
  </Dataset>
</GrapherWebService>
}}}
The elements have the following meanings:
 - {{{DisplayIntervalStart}}} and {{{DisplayIntervalEnd}}} determine the interval for which the data will be plotted, and only those station features will be listed which have been collected during the whole span of this interval.
 - {{{ServedFromURL}}} is the network path to the csmlGrapher service (''not'' just the Pylon)
 - Each {{{Dataset}}} element stands for one WFS acting as a source of data. 
   - {{{GeoServerURL}}} contains the URL of the WFS/GeoServer, used as a base of the GetFeatureInfo call.
   - {{{DatetimeQueryFormat}}} specifies which date format to specify date intervals in queries. 
   - {{{IsCSMLLinkVisible}}} (use values {{{yes}}} or {{{no}}}) specifies whether to show the feature query URLs as links in the station feature listing. 

== Source code notes ==

This section describes how source code is organised. 
The two 

=== Development notes for {{{csml2kml}}}  ===

The egg's {{{csml2kml}}} directory contains all Python modules, and the following 4 sub-directories:
 - {{{scripts}}} - contains Python stand-alone scripts, runnable from command line, which both use a config file as a means of input
 - {{{tests}}} - contains tests, which simply print output (rather than fail/pass messages), and are runnable as {{{python test_filename.py}}}, but please look at the contents first to see what the test does
 - {{{config}}} - a preferred location for config files for the scripts, in XML format, which define locations of used Web services etc.
 - {{{output}}} - empty after installation, optionally usable for output of scripts and tests

The modules...

There is a naming convention that wrappers of entities which normally exist as XML elements are prefixed with XML spacename. For instance, a class WMSLayer is a wrapper for an element ''<wms:Layer>''. Wrappers for KML elements (in {{{kml}}} namespace), which are used for output, contain a {{{build()}}} method, which builds each instance into an {{{ElementTree.Element}}} object. On the contrary, wrappers for elements in {{{wms}}} and {{{np}}} namespaces contain {{{parseXML()}}} method, as they are used for input.

=== Development notes for {{{csml2kmlpylon}}}  ===