Version 47 (modified by domlowe, 13 years ago) (diff) |
---|
The NERC Portals project
Since it's inception the NERC Portals project has started to be called the Data Portals Prototype Project (DPPP) across the participating institutions.
See
- CedaDevEnvironment being created on bond.
Conversion from CSML to KML: User manual
Since CSML currently lacks the constructs for describing the context for the CSML features, the actual mechanisms used to convert data from CSML to KML do use limited amount of CSML as a data source. The rest comes from different languages derived from GML. Hence this project became rather a demonstration of how OGC web services can be integrated into a common framework, which has a potential of working using purely CSML data, once CSML matures enough to contain all information in it.
The functionality is split in two:
- csml2kml (an egg) - acts as an API, but accompanied by command-line runnable scripts. Provides static CSML-to-KML conversion functionality, with the output being KML documents stored in files (*.kml, *.kmz) or directory hierarchies of files connected via KML NetworkLink's.
- csml2kmlpylon (a Pylon application) - is a web server (to be run by the BADC) serving a csmlGrapher web service that renders dynamic HTTP content on top of static data in KML files being viewed in Google Earth.
Installation on a target machine
Both csml2kml and csml2kmlpylon require matplotlib, cElementTree and csml.
Installation of csml2kml
The procedure is:
- Use the standard easy_install command to install latest version of the egg on the target machine.
easy_install -f http://ndg.nerc.ac.uk/dist csml2kml
This will install the egg and also install two scripts (called wms2kml and stations2kml).
Temporary storage and archival of KML results
For daily use as a directory for KML output, it is recommended to use the following directory with svn:ignore='*' set on it: http://proj.badc.rl.ac.uk/ndg/browser/DPPP/kml/kml-output
There is now a directory in the repository: http://proj.badc.rl.ac.uk/ndg/browser/DPPP/kml/data-archive usable as space for storing important generated results as tarballs.
To re-generate the KML files you would need to do the following:
- Check out the below *.conf.xml files from the repository at :
svn+ssh://proj.badc.rl.ac.uk/disks/glue1/svn-repository/ndg/DPPP/kml/python/csml2kml/csml2kml/config See further below for format details.
- Edit the output destination directories in them and make sure you have created the output directories.
- Run the following commands (as of 2 April 2008, these are the existing datasets):
wms2kml ar4.wms2kml.conf.xml wms2kml obs.wms2kml.conf.xml stations2kml midas.csml2kml.conf.xml stations2kml ecn.csml2kml.conf.xml
Installation of csml2kmlpylon
- Check out the Pylon application from the repository, at Subversion URL:
svn+ssh://proj.badc.rl.ac.uk/ndgsvn/DPPP/kml/python/csml2kmlpylon
- Edit the config file, csml2kmlpylon/config/csml2kmlpylon.conf.xml (relative to working copy root). See section Usage of the {{{csml2kmlpylon}}} web service for details on the config file.
- If the KML/KMZ files pointing to this web service are not yet served from elsewhere, you can put them (or symlinks to them) in the csml2kmlpylon/public directory.
- Run the service by executing:
paster serve deploy.ini --deamon
Alternatively, if you want to do development on the Pylon (with Pylon debug mode turned on), run the command:paster serve development.ini --deamon
Usage of the csml2kml scripts
The wms2kml.py script
Run this script as:
python wms2kml.py filename.conf.xml
where filename.conf.xml stands for an XML config file.
The script interrogates a WMS web service using the GetCapabilities call and thus acquires a <wms:Capabilities> XML document from it, which contains description of WMS layers served by the WMS web service. Then, it exports this into a hierarchy of embedded directories forming a structure which contains a hierarchy of expandable KMZ documents linked together using the <kml:NetworkLink> tag, and loaded on demand. The files can then be served to the outer world via HTTP.
An example format of the config file is as follows (example config files can be found in the egg's csml2kml/config sub-directory):
<WMSLayers2KML> <WMSRequest> <URL>http://www-devel.ipcc-data.org/maps/wms/obs</URL> <ServiceVersion>1.1.1</ServiceVersion> <ImageFormat>image/png</ImageFormat> <ImageWidth>960</ImageWidth> <ImageHeight>480</ImageHeight> <CRS>EPSG:4326</CRS> </WMSRequest> <OutputRootDirectory>/home/users/mkochan/svn-root/kml/csml2kml/output/wms-obs2</OutputRootDirectory> <ServerRootDirectory>http://bond.badc.rl.ac.uk:8089</ServerRootDirectory> </WMSLayers2KML>
The field URL is a base URL of the WMS web service serving the intended dataset.
The stations2kml.py script
Run this script as:
python stations2kml.py filename.conf.xml
where filename.conf.xml stands for an XML config file.
This script interrogates a WFS to get a list of (meteorological) stations, recorded using an element <np:Station> defined for the purposes of this project. It generates a KMZ file that contains a number of <kml:Placemark> elements, each standing for a single station. Each station has has a balloon filled according to a defined template, which refers to a URL of the running csml2kmlpylon pylon. Thus dynamic content about each station can be presented.
An example config file looks as follows:
<Stations2KML> <DocumentName>MIDAS stations</DocumentName> <UseRegions>yes</UseRegions> <GetAllStationsRequestURL><![CDATA[http://bond.badc.rl.ac.uk:8084/geoserver/wfs?request=getFeature&service=wfs&version=1.1.0&typename=np:Station&maxFeatures=100000]]></GetAllStationsRequestURL> <BalloonTemplate><h2>Station $[station_name]</h2>Click here to see the list of <a href="http://superglue.badc.rl.ac.uk:8084/csmlGrapher/list?dataset_id=midas&station_name=$[station_name]">CSML features associated with this station</a> (this will open a window of your default web browser).</BalloonTemplate> <StationData> <Datum name="station_id">#ID#</Datum> <Datum name="station_name">#NAME#</Datum> </StationData> <DisplayIntervalStart>2006-6-1</DisplayIntervalStart> <DisplayIntervalEnd>2006-8-1</DisplayIntervalEnd> <OutputKmzFilename>/home/users/mkochan/svn-root/kml/csml2kml/output/midas_stations.kmz</OutputKmzFilename> </Stations2KML>
Details for some non-obvious settings are as follows:
- UseRegions - determines whether to use embedding into regions.
- If set to no, stations will be recorded simply as a list of KML placemarks, and will be visible from all altitudes. May cause slow rendering for large number of placemarks.
- If set to yes, stations will be divided by recursively splitting the the covered are into 4 smaller KML regions. The actual station KML placemarks will be placed at the bottom of the hierarchy. Therefore, they will be forced to be visible only for closer zooms onto the ground, so rendering will be quick.
- GetAllStationsRequestURL - is an URL at which the <np:Station> list can be retrieved
- BalloonTemplate - is a template that is used for the balloon of every station/placemark. It should contain an URL (as part of an HTML anchor) linking to the csml2kmlpylon Web service.
- StationData - contains station-specific data that are substituted in the balloon template. The #ID# and #NAME# fields get replaced by the script with station identifier (such as "midas.station.2500") and name (such as "FAIR ISLE"), respectively. Note that a station's latitude and longitute need not be shown in the balloon, as they can be found by right-clicking on the station placemark.
- DisplayIntervalStart and DisplayIntervalEnd specify the time internal, which is used in marking some stations red: the stations that contain at least one CSML feature which has been measured during this time interval, will be painted red; the rest will be white.
Usage of the csml2kmlpylon web service
Assuming that the pylon is served at URL http://server:port, the web service can be used in two ways:
- Using the list sub-service to provide a listing of CSML features available for a given station:
http://server:port/csmlGrapher/list?dataset_id=DATASET_ID&station_name=STATION%20NAME
The "dataset" here refers to an URL of a WFS server from which station data can be fetched. Each station is identified by its name (e.g. "FAIR ISLE", rather than station ID (e.g. "midas.station.3"); this is due to how our GeoServer? identifies the stations. - Using the plot sub-service to plot a specific CSML feature of a specific station:
http://server:port/csmlGrapher/list?dataset_id=DATASET_ID&station_name=STATION%20NAME&feature_id=feature_id
The HTML produced by csmlGrapher's list sub-service produces links containing the corresponding plot sub-service URLs and optionally also verbatim links to WFS that return underlying CSML if the user has access to that WFS.
Please make sure you use Firefox to view the generate web content (possibly by choosing to use an external viewer in Google Earth's Tools/Options? menu) as IE7 does not show the SQL errors properly.
The pylon currently contains only a single controller called csmlGrapher.py.
After being installed, the server can be run by the command:
paste serve $PATH_TO_CSML2KMLPYLON/deploy.ini
The controller csmlGrapher.py also has a config file which needs to be specified in development.ini in the [app:main] section e.g.:
... [app:main] ... csmlGrapher.configfile = %(here)s/csml2kmlpylon/config/csml2kmlpylon.conf.xml ...
Format of the config file is as follows:
<GrapherWebService> <DisplayIntervalStart>2006-6-1</DisplayIntervalStart> <DisplayIntervalEnd>2006-7-1</DisplayIntervalEnd> <ServedFromURL>http://superglue.badc.rl.ac.uk:8084/csmlGrapher</ServedFromURL> <Dataset id="midas" name="MIDAS"> <GeoServerURL>http://bond.badc.rl.ac.uk:8084/geoserver/wfs</GeoServerURL> <DatetimeQueryFormat>1-JAN-1970</DatetimeQueryFormat> <IsCSMLLinkVisible>yes</IsCSMLLinkVisible> </Dataset> <Dataset id="ecn" name="ECN"> <GeoServerURL>http://dppp.nerc-lancaster.ac.uk:8080/geoserver/wfs</GeoServerURL> <DatetimeQueryFormat>UTC_TIMESTAMP</DatetimeQueryFormat> <IsCSMLLinkVisible>yes</IsCSMLLinkVisible> </Dataset> </GrapherWebService>
The elements have the following meanings:
- DisplayIntervalStart and DisplayIntervalEnd determine the interval for which the data will be plotted, and only those station features will be listed which have been collected during the whole span of this interval.
- ServedFromURL is the network path to the csmlGrapher service (not just the Pylon)
- Each Dataset element stands for one WFS acting as a source of data.
- GeoServerURL contains the URL of the WFS/GeoServer, used as a base of the GetFeatureInfo? call.
- DatetimeQueryFormat specifies which date format to specify date intervals in queries.
- IsCSMLLinkVisible (use values yes or no) specifies whether to show the feature query URLs as links in the station feature listing.
Source code notes
This section describes how source code is organised.
Source code notes for csml2kml
The egg's csml2kml directory contains all Python modules, and the following 3 sub-directories:
- scripts - contains Python stand-alone scripts, runnable from command line, which both use a config file as a means of input
- tests - contains tests, which simply print output (rather than fail/pass messages), and are runnable as python test_filename.py, but please look at the contents first to see what the test does
- config - a preferred location for config files for the scripts, in XML format, which define locations of used Web services etc.
The EpyDoc for the modules is available at http://glue.badc.rl.ac.uk/doc/dppp/epydoc/csml2kml/index.html.
There is a naming convention that wrappers of entities which normally exist as XML elements are prefixed with XML spacename. For instance, a class WMSLayer is a wrapper for an element <wms:Layer>. Wrappers for KML elements (in kml namespace), which are used for output, contain a build() method, which builds each instance into an ElementTree.Element object. On the contrary, wrappers for elements in wms and np namespaces contain parseXML() method, as they are used for input.
Source code notes for csml2kmlpylon
The EpyDoc for the whole Pylon is available at http://glue.badc.rl.ac.uk/doc/dppp/epydoc/csml2kmlpylon/index.html (especially look at controllers.csmlGrapher).