wiki:CEDAWPS

Version 6 (modified by astephen, 9 years ago) (diff)

--

CEDA Web Processing Service

This page has been set up to record the status and working of the CEDA WPS. Which builds upon the  WPS. This page includes:

About The CEDA WPS

The CEDA WPS runs at:

It has a WPS and WPS User Interface running on ceda-wps1.badc.rl.ac.uk as user cwps.

Jobs are scheduled on ceda-batch1.badc.rl.ac.uk using SGE.

Both of these Virtual Machines currently run on kona.badc.rl.ac.uk.

The Source Code

The Source Code is kept at:

Admin Interface

There is an administrator interface at:

You can be logged in as the administrator as well as a CEDA user, they use different systems to authenticate.

Jobs page

The jobs page lists current and old jobs. It can be modified to list what ever the user (or administrator) wants, see:

There are filters on the page or you can construct URLs such as:

TO DO List

The deployment environment TO DOs:

  1. Define a method of copying the VMs
  2. Set up a second server holding WPS and Batch VMs (v2)
  3. Load-balance ceda-wps1 behind ceda-wps before release
  4. Create a sensible test environment separate from the release version

WPS (back-end) TO DOs:

  1. Add "CANCELLED" status to WPS instead of it defaulting to "FAILED".
  2. Remove DCIP-relevant tests and work out how to add tests for local processes (in proc_tests dir).
  3. Need to generate the NDG-security policy.xml file from a script. This would allow quick adding of new datasets and their required code for download/status/jobs URLs (etc.). See #IdeasPolicyXML below.

WPS UI (front-end) TO DOs:

  1. Jobviewer controller doesn't work for FAILED jobs. It should.
  2. When a job has been CANCELLED the UI should stop polling (on the jobviewer page).
  3. Should there be polling (or at least re-load) on the jobs page?
  4. Can we get pylons interactive debugger working with the multi-processed secured service under apache?
  5. Recenter box to move hand and move icons across so visible when bbox is smaller.

Process-specifice TO DOs:

  1. When Dry Run is run then it estimates number of src_ids. If this allows then actual number is not allowed then an error is produced by non-dry run. Need to catch this in the UI.
  2. Need a generic catcher for responses from WPS so that errors are easily understood by interface.
  3. Select a stations file from GetUKWeatherStations then also select some counties in extract UK Data submit page. Then see why job fails.

Suggestions List

Please add your own suggestions below.

  • If you go through to submitting a job it takes you to the OpenID sign on.. that’s good... however, I enter my details and as I don't have the required roll it gave me the unauthorised page with a link to the CEDA site, but no way of navigating back to the WPS home page.

AS: Needs looking into.

  • Looking at the jobs list it might be better to note that "access denied" rather than "failed" for jobs where I couldn't have access to a process - otherwise we'll end up mis-reporting these instances. Also, incorrect access would be best sending the user to a page giving link to apply for appropriate permissions.

AS: At present the status also shows as FAILED if the job was CANCELLED, see item on TODO list.

Ideas for fixes

Ideas Policy XML

Need a script to create the "policy.xml" file, requires 2 configs:

$ cat roles.ini
[DEFAULT]
surface = 1
link = 2
ecmwf_op = 3

$ cat policy_rules.ini
[DEFAULT]
GetUKWeatherStations = surface
NCDumpIt = ecmwf_op
DoubleIt = 

See my notes for code.