wiki:Discovery/DiscoveryWebServiceMEDIN

Version 59 (modified by sdonegan, 9 years ago) (diff)

--

NERC DataGrid Discovery Web Service : MEDIN Improvements November 2009

The NERC DataGrid (NDG) Discovery Web Service provides a search interface to metadata records harvested from collaborating data providers and is the backend server to which the NERC Data Discovery Service is a client.

Introduction

The Discovery Web Service is a presentation-free web service which acts as a search engine on top of the NDG Discovery metadata catalogue. This catalogue is dynamically populated by the harvesting of metadata (by the  CEDA group at RAL) from a number of collaborating data providers, who make their metadata available in one of a number of supported formats.

The search capability provided by the service enables full-text and spatio-temporal searches of catalogued metadata records and returns search results a defined XML structure, enabling search clients to be constructed by interested parties for their own purposes. The NERC Data Discovery Service is one such client, as is the  Environmental Data Portal, among other examples.

Scope

This documentation covers functionality proposed for the improvements commissioned by MEDIN. Information on reporting and progress can be found  here.

Connectivity

Consumers may access the discovery service via SOAP. Client implementations should be generated from the WSDL at the following URIs:

(URIs tbc pending implementation of proposed improvements)

XML Data Types

The XML documents used as request and response messages for each of the service operations (methods) are defined in the XML schema section of the WSDL document. The structure of each of these messages is discussed as part of the operation/method descriptions below. Automatically-generated documentation for the schema showing the full structure is available here Download.

Discovery Service Operations

The discovery service implements 4 operations, namely:

  • getListNames
  • getList
  • doSearch
  • doPresent

getListNames operation

The discovery web service relies on several lists of valid terms which are specific to the functionality of this service. The reason for using these 2 "helper" operations rather than encoding these valid terms as <xs:enumeration> in the schema part of the WSDL, is so that future modifications to the service need not necessarily require the modification of the WSDL (which can be incovenient for clients already developed around a particular release of the WSDL). The getListNames operation simply returns the names of these lists, which can then be used in a subsequent call to the getList operation.

The WSDL document defines the getListNamesRequest message as an empty <GetListNames> element, so the request message should look like this (omitting the SOAP Envelope & Body parent elements):

<m:GetListNames xmlns:m="urn:DiscoveryServiceAPI"/>

The getListNamesResponse message comprises a <GetListNamesReturn> element, with child elements containing the names of the lists available for inspection:

<sch:GetListNamesReturn xmlns:sch="http://medin.discovery.services.ndg/schema">
			<sch:listNames>
				<sch:listName>TermTargetList</sch:listName>
				<sch:listName>PresentFormatList</sch:listName>
				<sch:listName>OrderByFieldList</sch:listName>
				<sch:listName>ScopeList</sch:listName>
				<sch:listName>SpatialOperatorList</sch:listName>
				<sch:listName>SpatialReferenceSystemList</sch:listName>
				<sch:listName>DateRangeTargetList</sch:listName>
				<sch:listName>TemporalOperatorList</sch:listName>
				<sch:listName>MetadataFormatList</sch:listName>
				<sch:listName>RecordDetailList</sch:listName>
			</sch:listNames>
		</sch:GetListNamesReturn>

getList operation

The contents of each of the lists named by the getListNames operation are accessible by invoking a call to the getList operation, with the name of the list as the single argument, encoded as a getListRequest message, as defined in the WSDL :

Request:

<m:GetList xmlns:m="http://medin.discovery.services.ndg/schema">
      <m:listName>PresentFormatList</m:listName>
</m:GetList>

Response:

<sch:GetListReturn xmlns:sch="http://medin.discovery.services.ndg/schema">
			<sch:list name="PresentFormatList">
				<sch:listMember>DC</sch:listMember>
				<sch:listMember>DIF_v9.4</sch:listMember>
				<sch:listMember>MEDIN_v2.3.1</sch:listMember>
			</sch:list>
		</sch:GetListReturn>

An explanation of the PresentFormatList is given later, in the context of the doPresent operation.

doSearch operation

The doSearch operation performs a search against the NDG discovery database. Queries to this database are formulated from the doSearchRequest message, and forwarded to the database via private methods (i.e. the consumer of the web service is not able directly to interact with the database).

Although outside the scope of the Discovery web service itself, it is worth explaining the structure of the NDG Discovery database which is searched by the service. This is populated from records harvested via  OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) from collaborating data providers. Records are harvested in the ISO19115-compliant metadata format agreed by the NERC Metadata Standards Working Group (MEDIN ISO v2.3.1) with legacy support for the previous format (DIF v9.4), and are tagged at ingest time with one or more "scope" keywords (listed in the scopeList list available from the getList operation). These enable the search to be restricted to particular communities, namely NERC, NERC_DDC (Designated Data Centres) and MEDIN (Marine Environmental Data Information Network). Limited quality control on ingested records is also applied at ingest time, but it is the responsibility of the data provider to ensure that metadata records are provided to sufficient quality to enable them to be visible in the system.

The doSearchRequest message is shown in schema form below (click image to enlarge):

Choice of search criteria: <TermSearch>, <SpatialSearch> and <TemporalSearch>

The <SearchCriteria> element acts as a container enabling the selection of one or more of <TermSearch>, <SpatialSearch> and <TemporalSearch>. Multiple <TermSearch> elements, but only one of each of <SpatialSearch> and <TemporalSearch> may be included, so that the search has at least one component. If more than one of these three main types is specified, the resulting search combines the components in "AND" combination (i.e. metadata records should match the term search AND the spatial search AND the temporal search criteria).

TermSearch

TermSearch is a full-text search invoked on a specific target field in the discovery database. Child elements <!Term> and <TermTarget> should be populated as follows:

  • <Term> : text term to search for. Whitespace separates component words, which are searched in "OR" combination unless the "+" symbol is used between them, in which case the words joined in this way are searched in "AND" combination.
  • <TermTarget> : target field name taken from the TermTargetList list of valid term targets.

If multiple <TermSearch?> elements are present (e.g. to search different <TermTarget>s) ), these are interpreted as successive term searches to be combined in "AND" combination. For example:

  <TermSearch>
    <Term>snow + rain</Term>
    <TermTarget>Abstract</TermTarget>
  </TermSearch>
  <TermSearch>
    <Term>lawrence</Term>
    <TermTarget>Author</TermTarget>
  </TermSearch>

means

  Search for records where:
    abstract contains "snow" AND "rain"
  AND
    author contains "lawrence"

If we were to extend the example by adding an additional <TermSearch> also targetted at the abstract, this would be combined in OR combination with the first <TermSearch> for that target (abstract), i.e.

  <TermSearch>
    <Term>snow + rain</Term>
    <TermTarget>Abstract</TermTarget>
  </TermSearch>
  <TermSearch>
    <Term>lawrence</Term>
    <TermTarget>Author</TermTarget>
  </TermSearch>
  <TermSearch>
    <Term>hail</Term>
    <TermTarget>Abstract</TermTarget>
  </TermSearch>

This would be interpreted to mean:

  Search for records where:
    abstract contains ("snow" AND "rain") OR "hail"
  AND
    author contains "lawrence"
Spatial searching : <SpatialOperator> and <BoundingBox>

The search may incorporate a spatial query to restrict results to those metadata records having spatial coverage(s) matching the search criteria defined by the <BoundingBox> elements <LimitNorth>, <LimitSouth>, <LimitEast> and <LimitWest>. An optional element <SpatialReferenceSystem> may be populated with an entry from the SpatialReferenceSystem list to specify an alternative spatial reference system (SRS) of the bounding box coordinates. (*Note : this feature is included for future development e.g. ability to supply spatial search coordinates in British National Grid coordinates. Currently the only supported SRS is EPSG:4326 (WGS84 lat/lon), and this will remain as the default if no SRS is specified.)

When using SRS EPSG:4326 (default), values for <LimitNorth>, <LimitSouth>, <LimitEast> and <LimitWest> should be given in decimal degrees latitude and longitude. <LimitNorth> and <LimitSouth> must be in the range -90.0 to +90.0, with <LimitNorth> greater than <LimitSouth>. <LimitWest> and <LimitEast> must be in the range -180.0 to 180.0 and <LimitEast> should be greater than <LimitWest>. Bounding boxes that span the -180 degree meridian, or the poles, are not supported.

An optional <SpatialOperator> may be included, populated with a term from the SpatialOperatorList, defining the comparison to be applied to spatial coverage(s) related to metadata records. The default value is "overlaps". Note that in the discovery index database, metadata records may contain several spatial coverages, so a match can occur if any of the spatial coverages related to the metadata item match the criteria specified in the spatial search.

Temporal searching : <DateRange>

The search may incoporate a temporal query to restrict results to those metadata records having temporal coverage(s) matching the search criteria specified within <DateRange>. One or two <Date> elements may be specified, to represent either a single date, or a date range. Each <Date> element must contain a <DateValue> element populated in the form YYYY-MM-DD, and optionally a <TemporalOperator> element, populated with a value from TemporalOperatorList, defining the semantic meaning of this date criterion in the search. In addition, an optional <DateRangeTarget> element may be included, populated with a value from the DateRangeTargetList, to enable searching of dates other than the default of "temporal coverage of data". Examples are shown below:

<DateRange>
  <Date>
    <DateValue>2001-01-01</DateValue>    
  </Date>
  <Date>
    <DateValue>2002-02-03</DateValue>    
  </Date>
</DateRange>

means

  Find metadata records where the temporal coverage(s) of the data
  overlaps the date range 2001-01-01 to 2002-02-03 inclusive.
<DateRange>
  <Date>
    <DateValue>2001-01-01</DateValue>
    <temporalOperator>Before</temporalOperator>    
  </Date>
  <Date>
    <DateValue>2002-02-03</DateValue>    
    <temporalOperator>After</temporalOperator>    
  </Date>
</DateRange>

means

Find metadata records where the temporal coverage(s) of the data
is outside of the date range 2001-01-01 to 2002-02-03 inclusive.
<DateRange>
  <Date>
    <DateValue>2001-01-01</DateValue>
    <temporalOperator>OnOrbefore</temporalOperator>    
  </Date>
</DateRange>
Find metadata records where the temporal coverage(s) of the data
is on or before the date 2001-01-01.
<DateRange>
  <Date>
    <DateValue>2001-01-01</DateValue>
    <temporalOperator>OnOrbefore</temporalOperator>    
  </Date>
  <DateRangeTarget>LastRevisionDate</dateRangeTarget>
</DateRange>

means

Find metadata records where the last revision date of the data
is on or before the date 2001-01-01.

Choice of Retrieve criteria: <OrderBy>, <RecordDetail> and <MetadataFormat>

Ordering

Ordering of the result set can optionally be requested by providing an <OrderBy> element containing one or more <OrderByField>s, each with an optional associated <OrderByDirection> (default : descending). Available fields for use as an <OrderByField> are listed in the OrderByFieldList.

RecordDetail?

The optional <RecordDetail> element enables selection of the level of detail included in each result returned in the search result. Values must be one of those available from the RecordDetailList. Default is DocumentId?, which simply returns the id of the document corresponding to the result. See Search Results section for further explanation of the structures returned in each of these cases.

MetadataFormat?

The optional <MetadataFormat?> element can be used to restrict the search to records in the discovery index whose original (harvested) representation was in the format specified. The format must be one of the values listed in the metadataFormatList available from the getList operation. Default behaviour if this element is omitted is not to restrict the search in this way (and return results irrepsective of their harvested format).

Paging : <Start> and <HowMany>

The optional elements <Start> and <HowMany> control which records from the result set should be returned (although the total number of hits is always returned as a number to aid with paging in clients). If <Start> is omitted, the default value used is 1 (i.e. the first record). If <HowMany> is omitted, all records are returned.

Scope of search: <Scope>

The optional <Scope> element can be used to restrict the search to onr or more of the supported NDG Data Provider Groups, defined in NDG controlled vocabulary  http://vocab.ndg.nerc.ac.uk/list/N010/0. Currently supported values from this vocabulary are these are given in the the ScopeList list accessible via the getList operation. If <Scope> is omitted, the search is not restricted in this way.

Search Results

The doSearchResponse message is defined in the WSDL as shown below (click image for larger version):

The <DoSearchReturn> element contains the following top-level elements:

Status
true if successful AND number of hits > 0, false otherwise (designed so that a client need only proceed to parse the rest of the message if results were successfully returned)
StatusMessage
Textual information regarding success / failure / errors
ResultId
reserved for future use
Hits
TOTAL number of hits returned
Documents
parent element for <DocumentId>, <DocumentBrief> OR <DocumentSummary> elements (as per choice in search request) containing returned results.

A result where no hits were returned is shown below:

<DoSearchReturn xmlns="urn:DiscoveryServiceAPI">
	<Status>false</Status>
	<StatusMessage>Search was successful but generated no results.</StatusMessage>
	<ResultId>0</ResultId>
	<Hits>0</hits>
</DoSearchReturn>

A result where 2 hits were returned, with the <RecordDetail> set to DocumentId, is shown below:

<DoSearchReturn xmlns="urn:DiscoveryServiceAPI">
	<Status>true</Status>
	<StatusMessage>Search was successful.</StatusMessage>
	<ResultId>0</ResultId>
	<Hits>2</Hits>
        <Documents>
          <DocumentId>idForResult1GoesHere</DocumentId>
          <DocumentId>idForResult2GoesHere</DocumentId>
        </Documents>
</DoSearchReturn>

If <DocumentBrief> is specified as the RecordDetail, a <DocumentBrief> element is returned for each result, as outlined in the doSearchResponeMessage, above. This contains the <DocumentId> element, containing the id of the document, but is accompanied by the additional element <Title>, containing the title from the metadata record, and a set of <OrderedField> elements corresponding to the <OrderByField>s used in the search request. In other words, the requested ordering fields are returned alongside the results so that a client can display the content of those fields which contributed to the resulting record ordering. The purpose of this <DocumentBrief> detail option is to enable clients to render a results list directly from the search response, without having immediately to invoke the doPresent operation to retrieve additional detail.

Similarly, if <DocumentSummary> is specified as the recordDetail, a <DocumentSummary> element is returned for each result, as outlined in the doSearchResponseMessage, above. In addition to the content added by the <DocumentBrief> option, <DocumentSummary> includes the metadata abstract, and temporal and spatial information. For the temporal and spatial components of this <DocumentSummary> the schema reuses the structures used for the search request, hence the optional temporalOperator are spatialOperator elements are redundant (and will be omitted) from the return context, however the dateRangeTarget element is useful as a contextual reminder of what the returned date pertains to (temporal coverage of data, last revision date of data, or ingestion date of metadata, etc.).

A corresponding <DocumentFull> structure is used in the doPresent operation operation as the structure in which the document payload is returned.

doPresent operation

The doPresent operation provides a means of retrieving (presenting) one or more XML documents from the database. The doPresentRequest message is defined as follows (click image to enlarge):

Within the <Documents> element, one or more <DocumentId> elements should each contain the name of a document (in the form returned in the <DocumentId> of the doSearchReturn message) to be retrieved. The optional <Format> element should be populated with one of the supported format names as listed by the PresentFormatList? accessible via the getList operation. All documents returned by a single invocation of the doPresent operation are returned in the same format, i.e. the choice of presentFormat applies to the doPresent request and not individual documents. For information, it should be noted that documents harvested (via OAI-PMH) are, at the time of ingest, stored in their native format but additionally converted to static copies of each of the supported return formats (listed in the PresentFormatList.) When a document is requested for presenation in a particular format, the static copy (either the native copy or the generated copy in the requested format) is returned from the database.

doPresent response

The doPresentResponse message is defined in the WSDL as follows (click image to enlarge):

The <DoPresentReturn?> element contains the following top-level elements:

<Status>
true if there are any documents returned in the payload, false otherwise.
<StatusMessage>
Textual information regarding success / failure / errors.
<Documents>
If some documents have been successfully returned, a <documents> element is present and will contain a child <DocumentFull> element for each document retrieved. In the case where some but not all documents are successfully returned, the <Documents> element will contain populated <DocumentFull> elements for the successfully-retrieved documents, but <DocumentFull> elements containing a populated <DocumentId> but unpopulated <Document> element for those where retrieval failed. If NO documents are successfully returned, however, then the <Status> is set to false and no <Documents> element is included in the doPresentResponse message.

The <Document> element, if present and populated, contains the retrieved document as an encapsulated string representation of the XML. Depending on the client used to display the payload document, it either appears contained within a <![CDATA[ ... ]]> construct, or as XML with the opening angle brackets "<" escaped as "&lt;". Most XML parsers should successfully parse the string to reconstruct the XML document, but it is returned in this form to avoid namespace issues.

The following request / response sequence shows a successful doPresent operation:

Request:

<m:DoPresent xmlns:m="urn:DiscoveryServiceAPI">
    <m:Documents>
        <m:DocumentId>badc.nerc.ac.uk__DIF__dataent_claus.xml</m:DocumentId>
    </m:Documents>
    <m:Format>DIF_v9.4</m:Format>
</m:DoPresent>

Response:

<DoPresentReturn xmlns="urn:DiscoveryServiceAPI">
    <Status>true</Status>
    <StatusMessage>Success</StatusMessage>
    <Documents>
        <DocumentFull>
           <DocumentId>badc.nerc.ac.uk__DIF__dataent_claus.xml</DocumentId>
           <Title>Cloud Archive User Service data (CLAUS)</Title>
           <Abstract>Global Brightness Temperature imagery from the 
Cloud Archive User Service project. This project produced a long 
time-series of global thermal infra-red imagery of the Earth using 
data from operational meteorological satellites, which was used in 
validating atmospheric General Circulation Models</Abstract>
           <Document>&lt;DIF xmlns="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
&lt;Entry_ID>badc.nerc.ac.uk:DIF:dataent_claus&lt;/Entry_ID>
 (.. truncated! ..) &lt;/DIF></Document>
        </DocumentFull>
    </Documents>
</DoPresentReturn>

Term Lists

The following lists and their contents are available via the getListNames and getList operations, but are repeated here with explanations of the list members.

TermTargetList

FullText
Target is a stored version of the original harvested XML metadata document
!Author
Target is the entry in the discovery index database corresponding to the author of the data described by the metadata.
!Parameter
Target is entries in the discovery index database corresponding to parameter keywords extracted from the metadata.
ResourceType
ResourceType? relates to MEDIN element 4
TopicCategory
TopicCategory? relates to MEDIN element 9
!Lineage
Lineage relates to MEDIN element 17
PublicAccessLimits
PublicAccessLimits? relates to MEDIN element 20 (Limitations on Public Access)
DataOriginator
DataOriginator? relates to MEDIN element 22.1
DataFormat
DataFormat? relates to MEDIN element 23

Note that for targeting a search at a specific date based element (i.e. MetadataUpdateDate?) the date must be entered in via the TemporalSearch? and the date range target specified as MetadataUpdateDate?

OriginalFormatList

DIF_v9.4
GCMD DIF format (version 9.4)
MEDIN_v2.3.1
Metadata format used by the Marine Environmental Data and Information Network, version 2.3.1
ISO19115
ISO19115 (Geographic Information: Metadata) encoded as ISO19139 XML. Currently an alias for MEDIN_v2.3.1, i.e. the MEDIN format is used as the implementation of ISO within the Discovery Service.

MetadataFormatList

DC
Dublin Core format
DIF_v9.4
GCMD DIF format (version 9.4)
MEDIN_v2.3.1
Metadata format used by the Marine Environmental Data and Information Network, version 2.3.1
ISO19115
ISO19115 (Geographic Information: Metadata) encoded as ISO19139 XML. Currently an alias for MEDIN_v2.3.1, i.e. the MEDIN format is used as the implementation of ISO within the Discovery Service.

OrderByFieldList

TextRelevance
Ranking metric based on relevance of match to search term (metric derived by postgres text ranking function).
DatasetStartDate
The start date of the date range given for the temporal coverage of the metadata record. Records with no start date defined are treated as if their start date is later than that last record with a start date defined, hence appearing at the end of the results.
DatasetEndDate
The end date of the date range given for the temporal coverage of the metadata record. Records with no end date defined are treated as if their end date is later than that last record with a start date defined, hence appearing at the end of the results.
DataCentre
The name of the data centre supplying the metadata record. In the case of records supplied in DIF format, this is the Data_Centre_Name/Short_Name field. In the case of other metadata formats, the most appropriate equivalent field is used as this index (e.g. "DistributorName?" for MDIP format)
DatasetResultsetPopularity
Measure of the popularity of a metadata record, related to how many times it has appeared in a result set in discovery searches.
!Proximity
The geographical proximity of the centre of the spatial coverage defined in the metadata record to the centre of the original search bounding box. Where no spatial information was originally selected proximity is calculated against the centre of a 'global' bounding box (0N, 0E). Metadata records with no spatial information originally defined are omitted from the search
ProximityNearMiss
An ordering based on records that were not within the originally requested spatial extent, but that occur within an arbritrary 10% of the original bounding box extent. Where records are present that satisfy this scenario, they are ordered according to proximity to the outside edge of the original bounding box. This option is to give users an idea of datasets that were close to the original bounding box and still matched the search without having to redefine the original bounding box and original search terms.
DatasetUpdateOrder
Date order based on the most recent update/edit by the original provider to the metadata record.
DatasetOrder
Alphabetical order based on the name of the metadata record. In DIF records this is the Entry_Title field and for MDIP the Title field.
DiscoveryIngestDate
Date order based on the date of insertion of that record into the underlying database supporting the service. Note that this differs from "datasetUpdateOrder" in that this records the actual date a record was placed in the database as opposed to the last edit date of that record.
!MEDINTermTarget.1*
One of fields specified by MEDIN for use as a termTarget : it should be possible to configure these for use as orderByFields, too.

ScopeList

MEDIN
Marine Data Information Partnership (organisation replacing MDIP in previous API versions)
NERC_DDC
NERC Designated Data Centres
NERC
NERC (General)
DPPP
Data Portals Project Provider

SpatialOperatorList

Overlaps (default)
Return metadata records having one or more spatial coverages that spatially overlap the search bounding box.
DoesNotOverlap
Return metadata records having no spatial coverages that spatially overlap the search bounding box.
Within
Return metadata records having one or more spatial coverages that are entirely within the search bounding box.

SpatialReferenceSystemList

EPSG:4326
WGS84 Lat/Lon?, in decimal degrees

DateRangeTargetList?

TemporalCoverage
The temporal coverage of the data described by the metadata record.
LastRevisionDate
The data of last revision to the dataset, recorded in the metadata record.
DiscoveryIngestDate
The date that the metadata record was (last) ingested into the discovery index.
MetadataUpdateDate
The date that the metadata record was updated by the originating data centre/data provider to the Discovery service.

TemporalOperatorList?

Equals
Target date value is the same as the date specified
DoesNotEqual
Target date values does not equal the date specified
OnOrBefore
Target date is equal to or before the date specified
OnOrAfter
Target date is equal to or after the date specified
Before
Target date is before the date specified
After
Target date is after the date specified

MetadataFormatList

(Note this is separate from presentFormatList, to distinguish between purposes). metadataFormatList lists values available to distinguish the original source format of a harvested metadata item.

DC
Dublin Core format
DIF_9.4
GCMD DIF format (version 9.4)
MEDIN_2.3
Metadata format used by the Marine Environmental Data and Information Network, version 2.3.1
ISO19115
ISO19115 (Geographic Information: Metadata) encoded as ISO19139 XML. Currently an alias for MEDIN_2.3, i.e. the MEDIN format is used as the implementation of ISO within the Discovery Service.

RecordDetailList

Id
Return a <DocumentId> element containing the id of the matching metadata document, in a form that can be used as input to a doPresent request.
Brief
Return a <DocumentBrief> element containing a <DocumentId> element, the title and ordering fields.
Summary
Return a <DocumentSummary> element containing content as per <DocumentBrief> but additionally abstract, temporal and spatial information.

Attachments