wiki:CowsWps/CDOWPSWorkingGroup/WPSAndWorkflows

Workflow management and the WPS

Introduction

Many potential usages of the WPS and CDO require us to develop tools to describe, interpret and execute workflows. Some example workflows might be:

  1. An external workflow application that makes a number of separate calls to one, or more, WPS instances. - External workflow management
  2. A WPS that can take a workflow description as its input (typically encoded in XML via HTTP POST) and can then execute a workflow that might involve a number of calls to local CDO and/or other software instances. - Internal workflow management with local services
  3. A WPS that can take a workflow description as its input and can then execute a workflow contacting both local processes and external services (with appropriate designated security implemented). - Internal workflow management with remote services

The three scenarios are characterised as:

  1. External workflow management
  2. Internal workflow management with local services
  3. Internal workflow management with remote services

1. External workflow management

If an external tool manages the workflow process then the WPS itself does not need to understand the concept of workflows internally. Once possible candidate for managing this is  VisTrails. It is a VisTrails? is a workflow management system. It is written in Python users can easily write own modules.

Gregory has written a simple module that can talk to a Web Processing Service. A workflow developer can transfer cdo commands through this interface to the COWS WPS. Parsing is done via the generic COWS WPS CDO module. Workflows are managed by the VisTrails? execution engine. They can be exported to XML for offline processing.

Another option to consider is  Amazon Simple Workflow Service (SWF). This can coordinate tasks inside or outside the Amazon cloud. There is a free tier which could be suitable for prototyping.

2. Internal workflow management with local services

This approach might be compatible with the use of VisTrails in the above external workflow management. It requires the WPS to understand a workflow description language encoded in XML. This could be the same XML export that VisTrails produces. We should consider this option.

3. Internal workflow management with remote services

This is the same case as (2) except that it includes calling out to external services and acting on the user's behalf. As a result it requires the delegated security services to be implemented. CEDA is currently developing this to run inside the COWS WPS.

This could also be compatible with VisTrails, but with the WPS invoking the VisTrails python libraries from within a process. This should be discussed more.