Version 75 (modified by sdonegan, 6 years ago) (diff)


NERC DataGrid Discovery Web Service : MEDIN Improvements November 2009

The NERC DataGrid (NDG) Discovery Web Service provides a search interface to metadata records harvested from collaborating data providers and is the backend server to which the NERC Data Discovery Service is a client.


The Discovery Web Service is a presentation-free web service which acts as a search engine on top of the NDG Discovery metadata catalogue. This catalogue is dynamically populated by the harvesting of metadata (by the  CEDA group at RAL) from a number of collaborating data providers, who make their metadata available in one of a number of supported formats.

The search capability provided by the service enables full-text and spatio-temporal searches of catalogued metadata records and returns search results a defined XML structure, enabling search clients to be constructed by interested parties for their own purposes. The NERC Data Discovery Service is one such client, as is the  Environmental Data Portal, among other examples.


This documentation covers functionality proposed for the improvements commissioned by MEDIN. Information on reporting and progress can be found  here.


Consumers may access the discovery service via SOAP. Client implementations should be generated from the WSDL at the following URIs:

(URIs tbc pending implementation of proposed improvements)

XML Data Types

The XML documents used as request and response messages for each of the service operations (methods) are defined in the XML schema section of the WSDL document. The structure of each of these messages is discussed as part of the operation/method descriptions below. Automatically-generated documentation for the schema showing the full structure is available here Download.

Discovery Service Operations

The discovery service implements 4 operations, namely:

  • getListNames
  • getList
  • doSearch
  • doPresent

getListNames operation

The discovery web service relies on several lists of valid terms which are specific to the functionality of this service. The reason for using these 2 "helper" operations rather than encoding these valid terms as <xs:enumeration> in the schema part of the WSDL, is so that future modifications to the service need not necessarily require the modification of the WSDL (which can be incovenient for clients already developed around a particular release of the WSDL). The getListNames operation simply returns the names of these lists, which can then be used in a subsequent call to the getList operation.

The WSDL document defines the getListNamesRequest message as an empty <GetListNames> element, so the request message should look like this (omitting the SOAP Envelope & Body parent elements):

<m:GetListNames xmlns:m="urn:DiscoveryServiceAPI"/>

The getListNamesResponse message comprises a <GetListNamesReturn> element, with child elements containing the names of the lists available for inspection:

<sch:GetListNamesReturn xmlns:sch="">

getList operation

The contents of each of the lists named by the getListNames operation are accessible by invoking a call to the getList operation, with the name of the list as the single argument, encoded as a getListRequest message, as defined in the WSDL :


<m:GetList xmlns:m="">


<sch:GetListReturn xmlns:sch="">
			<sch:list name="PresentFormatList">

An explanation of the PresentFormatList is given later, in the context of the doPresent operation.

doSearch operation

The doSearch operation performs a search against the NDG discovery database. Queries to this database are formulated from the doSearchRequest message, and forwarded to the database via private methods (i.e. the consumer of the web service is not able directly to interact with the database).

Although outside the scope of the Discovery web service itself, it is worth explaining the structure of the NDG Discovery database which is searched by the service. This is populated from records harvested via  OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) from collaborating data providers. Records are harvested in the ISO19115-compliant metadata format agreed by the NERC Metadata Standards Working Group (MEDIN ISO v2.3.1) with legacy support for the previous format (DIF v9.4), and are tagged at ingest time with one or more "scope" keywords (listed in the scopeList list available from the getList operation). These enable the search to be restricted to particular communities, namely NERC, NERC_DDC (Designated Data Centres) and MEDIN (Marine Environmental Data Information Network). Limited quality control on ingested records is also applied at ingest time, but it is the responsibility of the data provider to ensure that metadata records are provided to sufficient quality to enable them to be visible in the system.

The doSearchRequest message is shown in schema form below (click image to enlarge):

Choice of search criteria: <TermSearch>, <SpatialSearch> and <TemporalSearch>

The <SearchCriteria> element acts as a container enabling the selection of one or more of <TermSearch>, <SpatialSearch> and <TemporalSearch>. Multiple <TermSearch> elements, but only one of each of <SpatialSearch> and <TemporalSearch> may be included, so that the search has at least one component. If more than one of these three main types is specified, the resulting search combines the components in "AND" combination (i.e. metadata records should match the term search AND the spatial search AND the temporal search criteria).


TermSearch is a full-text search invoked on a specific target field in the discovery database. TermSearch elements must have an attribute named "id" with an integer value representing the order to be considered in complex multi termed searches, usually though "id" can be considered as the number of the TermSearch element. If more than one TermSearch element is used then all subsequent TermSearch elements (excluding the first) must also have an "operator" attribute describing the relationship between the TermSearch element in question with preceding TermSearchs (the order of these is given by the "id" attribute). Acceptable values for the operator attribute are "AND", "OR" and "NOT" which are to be considered reserved words (for an exception see "Exact Phrase Search" paragraph below).

Child elements <!Term> and <TermTarget> should be populated as follows:

  • <Term> : text term to search for. Whitespace separates component words, which are searched in "OR" combination unless the "+" symbol is used between them, in which case the words joined in this way are searched in "AND" combination. The text "AND" may also be subsituted for "+".
  • <TermTarget> : target field name taken from the TermTargetList list of valid term targets.

An example of a single TermSearch targeted at the FullText term target:

    <m:Term>snow + rain</m:Term>


  Search for records where:
    Full text contains "snow" AND "rain"

Other example of correct queries are

  <m:Term>snow rain</m:Term>     ---> "snow OR rain"

  <m:Term>snow AND rain</m:Term> ---> "snow AND rain"

  <m:Term>snow + - rain</m:Term> ---> "snow AND NOT rain"

  <m:Term>snow-rain</m:Term>     ---> "snow-rain"

Examples of incorrect queries are

  <m:Term>- rain</m:Term>

  <m:Term>snow - rain</m:Term>

  <m:Term>snow OR ! rain</m:Term> ---> Not supported

We can extend the above example by adding an additional <TermSearch> targeted at the Authors field:

  <m:TermSearch id="1">
    <m:Term>snow + rain</m:Term>
  <m:TermSearch operator="AND" id="2">


  Search for records where:
    abstract contains "snow" AND "rain"
    author contains "lawrence"

We can add further TermSearches to extend this query:

  <m:TermSearch id="1">
    <m:Term>snow + rain</m:Term>
  <m:TermSearch operator="AND" id="2">
  <m:TermSearch operator="OR" id="3">

This would be interpreted to mean:

Search for records where:
    abstract contains ("snow" AND "rain") OR "hail"
    author contains "lawrence"

..and so on.

Exact Phrase Search

If an "exact phrase" search is needed the sentence has to be wrapped with two double quotes. Consequently if the user is looking for looking for "Snow and rain" should be translated to Snow and rain

<m:Term>'''Snow and rain'''</m:Term>
Spatial searching : <SpatialOperator> and <BoundingBox>

The search may incorporate a spatial query to restrict results to those metadata records having spatial coverage(s) matching the search criteria defined by the <BoundingBox> elements <LimitNorth>, <LimitSouth>, <LimitEast> and <LimitWest>. An optional element <SpatialReferenceSystem> may be populated with an entry from the SpatialReferenceSystem list to specify an alternative spatial reference system (SRS) of the bounding box coordinates. (*Note : this feature is included for future development e.g. ability to supply spatial search coordinates in British National Grid coordinates. Currently the only supported SRS is EPSG:4326 (WGS84 lat/lon), and this will remain as the default if no SRS is specified.)

When using SRS EPSG:4326 (default), values for <LimitNorth>, <LimitSouth>, <LimitEast> and <LimitWest> should be given in decimal degrees latitude and longitude. <LimitNorth> and <LimitSouth> must be in the range -90.0 to +90.0, with <LimitNorth> greater than <LimitSouth>. <LimitWest> and <LimitEast> must be in the range -180.0 to 180.0 and <LimitEast> should be greater than <LimitWest>. Bounding boxes that span the -180 degree meridian, or the poles, are not supported.

An optional <SpatialOperator> may be included, populated with a term from the SpatialOperatorList, defining the comparison to be applied to spatial coverage(s) related to metadata records. The default value is "overlaps". Note that in the discovery index database, metadata records may contain several spatial coverages, so a match can occur if any of the spatial coverages related to the metadata item match the criteria specified in the spatial search.

Temporal searching : <DateRange>

The search may incoporate a temporal query to restrict results to those metadata records having temporal coverage(s) matching the search criteria specified within <DateRange>. One or two <Date> elements may be specified, to represent either a single date, or a date range. Each <Date> element must contain a <DateValue> element populated in the form YYYY-MM-DD, and optionally a <TemporalOperator> element, populated with a value from TemporalOperatorList, defining the semantic meaning of this date criterion in the search. In addition, an optional <DateRangeTarget> element may be included, populated with a value from the DateRangeTargetList, to enable searching of dates other than the default of "temporal coverage of data". Examples are shown below:



  Find metadata records where the temporal coverage(s) of the data
  overlaps the date range 2001-01-01 to 2002-02-03 inclusive.


Find metadata records where the temporal coverage(s) of the data
is outside of the date range 2001-01-01 to 2002-02-03 inclusive.
Find metadata records where the temporal coverage(s) of the data
is on or before the date 2001-01-01.


Find metadata records where the last revision date of the data
is on or before the date 2001-01-01.

Choice of Retrieve criteria: <OrderBy>, <RecordDetail> and <MetadataFormat>


Ordering of the result set can optionally be requested by providing an <OrderBy> element containing one or more <OrderByField>s, each with an optional associated <OrderByDirection> (default : descending). Available fields for use as an <OrderByField> are listed in the !OrderByFieldList.


The optional <RecordDetail> element enables selection of the level of detail included in each result returned in the search result. Values must be one of those available from the !RecordDetailList. Default is DocumentSimple, which simply returns the id of the document corresponding to the result. See Search Results section for further explanation of the structures returned in each of these cases.


The optional MetadataFormat element enables selection of the XML format returned in the DoSearch return DocumentFull/Document element. The default behavior is to return MEDIN_2.3 format metadata if MetadataFormat is not used (but RecordDetail is set to DocumentFull). Acceptable values for MetadataFormat can be found in the !MetadataFormatList

Paging : <Start> and <HowMany>

The optional elements <Start> and <HowMany> control which records from the result set should be returned (although the total number of hits is always returned as a number to aid with paging in clients). If <Start> is omitted, the default value used is 1 (i.e. the first record). If <HowMany> is omitted, all records are returned.

Scope of search: <Scope>

The optional <Scope> element can be used to restrict the search to onr or more of the supported NDG Data Provider Groups, defined in NDG controlled vocabulary Currently supported values from this vocabulary are these are given in the the ScopeList list accessible via the getList operation. If <Scope> is omitted, the search is not restricted in this way.


The optional <MetadataFormat> element can be used to restrict the search to records in the discovery index whose original (harvested) representation was in the format specified. The format must be one of the values listed in the !OriginalFormatList available from the getList operation. Default behaviour if this element is omitted is not to restrict the search in this way (and return results irrepsective of their harvested format).

Search Results

TheDoSearchResponse message is defined in the WSDL as shown below (click image for larger version):

The <DoSearchReturn> element contains the following top-level elements:

true if successful AND number of hits > 0, false otherwise (designed so that a client need only proceed to parse the rest of the message if results were successfully returned)
Textual information regarding success / failure / errors
reserved for future use
TOTAL number of hits returned
parent element for <DocumentSimple>, <DocumentBrief>, <DocumentSummary> and <DocumentFull> elements (as per choice in <RecordDetail> within search request) containing returned results.

A result where no hits were returned is shown below:

<sch:DoSearchReturn xmlns:sch="">
			<sch:StatusMessage>Search was successful but generated no results</sch:StatusMessage>

A result where 2 hits were returned, with the <RecordDetail> set to DocumentSimple, is shown below:

<sch:DoSearchReturn xmlns:sch="">

If <DocumentBrief> is specified as the RecordDetail, a <DocumentBrief> element is returned for each result, as outlined in the doSearchResponeMessage, above. This contains the <DocumentId> element, containing the id of the document, but is accompanied by the additional element <Title>, containing the title from the metadata record, and a set of <OrderedField> elements corresponding to the <OrderByField>s used in the search request. Within <OrderedField> are 2 elements, <MetricName> which is the name of the ordering field and <MetricValue> which is the actual value by which this document is ordered. These ordering fields are returned alongside the results so that a client can display the content of those fields which contributed to the resulting record ordering. The <AdditionalInformation> element holds child elements containing further detailed information (as requested) relating to the record in question. Where multiple values are associated with this element field within the database they are separated by a ";". If there is no information available to fill these fields then they are left blank. The purpose of this <DocumentBrief> detail option is to enable clients to render a results list directly from the search response, without having immediately to invoke the doPresent operation to retrieve additional detail.

Similarly, if <DocumentSummary> is specified as the recordDetail, a <DocumentSummary> element is returned for each result, as outlined in the doSearchResponseMessage, above. In addition to the content added by the <DocumentBrief> option, <DocumentSummary> includes the metadata abstract, and temporal and spatial information. For the temporal and spatial components of this <DocumentSummary> the schema reuses the structures used for the search request, hence the optional temporalOperator are spatialOperator elements are redundant (and will be omitted) from the return context, however the dateRangeTarget element is useful as a contextual reminder of what the returned date pertains to (temporal coverage of data, last revision date of data, or ingestion date of metadata, etc.).

Correspondingly the <DocumentFull> element builds upon the <DocumentBrief> and <DocumentSummary> elements adding further information (including <AdditionalInformation>) but importantly a <Document> element contains a CDATA representation of the original XML metadata with format corresponding to that chosen in <MetadataFormat> (defaulting to MEDIN_2.3). For information, it should be noted that documents harvested (via OAI-PMH) are, at the time of ingest, stored in their native format but additionally converted to static copies of each of the supported return formats (listed in the PresentFormatList.) When a document is requested for presenation in a particular format, the static copy (either the native copy or the generated copy in the requested format) is returned from the database.

These 4 document types comprise a data type "Documents" which is dependant on the values supplied in <RetrieveCriteria> and is also reused in the DoPresent operation.

DoPresent operation

The DoPresent operation provides a means of retrieving (presenting) one or more XML documents from the database. The DoPresent is distinguished from the DoSearch in that is allows the specific request of documents according to ID with the DoPresent response matching the order defined in the DoPresent request. The structure of the DoPresent is closely related to that of the DoSearch as most of the data types are reused.

. The doPresentRequest message is defined as follows (click image to enlarge):

Within the <Documents> element, one or more <DocumentId> elements should each contain the name of a document (in the form returned in the <DocumentId> of the DoSearchReturn message) to be retrieved. The <RetrieveCriteria> element again governs the level of detail and the XML format returned in the subsequent DocumentFull//Document element. The optional <OrderBy> element is ignored when <RetrieveCriteria> is used in a DoPresent request. All documents returned by a single invocation of the doPresent operation are returned in the same format, i.e. the choice of presentFormat applies to the doPresent request and not individual documents. If <MetadataFormat> is not defined then format defaults to MEDIN_2.3.

DoPresent response

The DoPresentResponse message is defined in the WSDL as follows (click image to enlarge):

The <DoPresentReturn?> element contains the following top-level elements:

true if there are any documents returned in the payload, false otherwise.
Textual information regarding success / failure / errors.
If some documents have been successfully returned, a <!Documents> element is present and consists of the same structure as the <!Documents> element in the DoSearchReturn element depending on the values entered in the <RetrieveCriteria> element in the DoPresent request.

The <Document> element, if present and populated, contains the retrieved document as an encapsulated string representation of the XML. Depending on the client used to display the payload document, it either appears contained within a <![CDATA[ ... ]]> construct, or as XML with the opening angle brackets "<" escaped as "&lt;". Most XML parsers should successfully parse the string to reconstruct the XML document, but it is returned in this form to avoid namespace issues.

The following request / response sequence shows a successful doPresent operation:


<m:DoPresent xmlns:m="urn:DiscoveryServiceAPI">


<DoPresentReturn xmlns="urn:DiscoveryServiceAPI">
           <Title>Cloud Archive User Service data (CLAUS)</Title>
           <Abstract>Global Brightness Temperature imagery from the 
Cloud Archive User Service project. This project produced a long 
time-series of global thermal infra-red imagery of the Earth using 
data from operational meteorological satellites, which was used in 
validating atmospheric General Circulation Models</Abstract>
           <Document>&lt;DIF xmlns="" 
 (.. truncated! ..) &lt;/DIF></Document>

Term Lists

The following lists and their contents are available via the getListNames and getList operations, but are repeated here with explanations of the list members.


Target is a stored version of the original harvested XML metadata document
Target is the entry in the discovery index database corresponding to the author of the data described by the metadata.
Target is entries in the discovery index database corresponding to parameter keywords extracted from the metadata.
ResourceType? relates to MEDIN element 4
TopicCategory? relates to MEDIN element 9
Lineage relates to MEDIN element 17
PublicAccessLimits? relates to MEDIN element 20 (Limitations on Public Access)
DataOriginator? relates to MEDIN element 22.1
DataFormat? relates to MEDIN element 27
AvailableDataFormats? relates to MEDIN element 23

Note that for targeting a search at a specific date based element (i.e. MetadataUpdateDate?) the date must be entered in via the TemporalSearch? and the date range target specified as MetadataUpdateDate?


Ranking metric based on relevance of match to search term (metric derived by postgres text ranking function).
The start date of the date range given for the temporal coverage of the metadata record. Records with no start date defined are treated as if their start date is later than that last record with a start date defined, hence appearing at the end of the results.
The end date of the date range given for the temporal coverage of the metadata record. Records with no end date defined are treated as if their end date is later than that last record with a start date defined, hence appearing at the end of the results.
The name of the data centre supplying the metadata record. In the case of records supplied in DIF format, this is the Data_Centre_Name/Short_Name field. In the case of other metadata formats, the most appropriate equivalent field is used as this index (e.g. "DistributorName?" for MDIP format)
Measure of the popularity of a metadata record, related to how many times it has appeared in a result set in discovery searches.
The geographical proximity of the centre of the spatial coverage defined in the metadata record to the centre of the original search bounding box. Where no spatial information was originally selected proximity is calculated against the centre of a 'global' bounding box (0N, 0E). Metadata records with no spatial information originally defined are omitted from the search
An ordering based on records that were not within the originally requested spatial extent, but that occur within an arbritrary 10% of the original bounding box extent. Where records are present that satisfy this scenario, they are ordered according to proximity to the outside edge of the original bounding box. This option is to give users an idea of datasets that were close to the original bounding box and still matched the search without having to redefine the original bounding box and original search terms.
Date order based on the most recent update/edit by the original provider to the metadata record.
Alphabetical order based on the name of the metadata record. In DIF records this is the Entry_Title field and for MDIP the Title field.
Date order based on the date of insertion of that record into the underlying database supporting the service. Note that this differs from "datasetUpdateOrder" in that this records the actual date a record was placed in the database as opposed to the last edit date of that record.
One of fields specified by MEDIN for use as a termTarget : it should be possible to configure these for use as orderByFields, too.


Marine Data Information Partnership (organisation replacing MDIP in previous API versions)
NERC Designated Data Centres
NERC (General)
Data Portals Project Provider


Overlaps (default)
Return metadata records having one or more spatial coverages that spatially overlap the search bounding box.
Return metadata records having no spatial coverages that spatially overlap the search bounding box.
Return metadata records having one or more spatial coverages that are entirely within the search bounding box.


WGS84 Lat/Lon?, in decimal degrees


The temporal coverage of the data described by the metadata record.
The data of last revision to the dataset, recorded in the metadata record.
The date that the metadata record was (last) ingested into the discovery index.
The date that the metadata record was updated by the originating data centre/data provider to the Discovery service.


Target date value is the same as the date specified
Target date values does not equal the date specified
Target date is equal to or before the date specified
Target date is equal to or after the date specified
Target date is before the date specified
Target date is after the date specified


Note this is separate from MetadataFormatList?, OriginalFormatList? lists values available to distinguish the original source format of a harvested metadata item.

GCMD DIF format (version 9.4)
Metadata format used by the Marine Environmental Data and Information Network, version 2.3
ISO19115 (Geographic Information: Metadata) encoded as ISO19139 XML. Currently an alias for MEDIN_v2.3.1, i.e. the MEDIN format is used as the implementation of ISO within the Discovery Service.


Note that MetadataFormatList? is used in the RetrieveCriteria? element to define the format of xml returned in either the DocumentFull? (doSearchReturnDocumentFullDocument element or doPresentReturnDocumentFullDocument element ). This is distinct from OriginalFormatList? which lists the accepted metadata input formats.

Dublin Core format
GCMD DIF format (version 9.4)
Metadata format used by the Marine Environmental Data and Information Network, version 2.3.1
ISO19115 (Geographic Information: Metadata) encoded as ISO19139 XML. Currently an alias for MEDIN_2.3, i.e. the MEDIN format is used as the implementation of ISO within the Discovery Service.


Return a <DocumentId> element containing the id of the matching metadata document, in a form that can be used as input to a doPresent request.
Return a <DocumentBrief> element containing a <DocumentId> element, the title and ordering fields.
Return a <DocumentSummary> element containing content as per <DocumentBrief> but additionally abstract, temporal and spatial information.