Ticket #107 (closed issue: wontfix)

Opened 13 years ago

Last modified 11 years ago

3rd Party Data Transfer - discussion required

Reported by: domlowe Owned by: spascoe
Priority: required Milestone: BETA
Component: ndg2 Version:
Keywords: Delivery Cc: selatham

Description (last modified by spascoe) (diff)

Issue: 3rd Party Data Transfer – is this an NDG2 requirement?

Assuming we need some sort of 3rd-party transfer I see 3 options:

  1. Live without dedicated 3rd-party transfer support in the DeliveryService. Server to server data transfer can be implemented on a per service basis using the DeliveryService in the standard client-server manner.
  2. Implement 3rd-party transfer as a high-level service which calls the low-level data transfer (presumably bbFTP) underneath. Such a service would act as the server to a controling client and a client to the other party to the data transfer.
  3. Implement 3rd-party transfer at the xFTP protocol level. See below for details about this.

Option 3.

Since GridFTP is supposed to do 3rd party data transfer I decided to take a look at the  GridFTP specification and was surprised to discover that the FTP protocol already supports 3rd party transfer in theory, it is just rarely implemented.

Here is the relevant extract from  RFC 959:

      When data is to be transferred between two servers, A and B (refer
      to Figure 2), the user-PI, C, sets up control connections with
      both server-PI's.  One of the servers, say A, is then sent a PASV
      command telling him to "listen" on his data port rather than
      initiate a connection when he receives a transfer service command.
      When the user-PI receives an acknowledgment to the PASV command,
      which includes the identity of the host and port being listened
      on, the user-PI then sends A's port, a, to B in a PORT command; a
      reply is returned.  The user-PI may then send the corresponding
      service commands to A and B.  Server B initiates the connection
      and the transfer proceeds.  The command-reply sequence is listed
      below where the messages are vertically synchronous but
      horizontally asynchronous:

         User-PI - Server A                User-PI - Server B
         ------------------                ------------------

         C->A : Connect                    C->B : Connect
         C->A : PASV
         A->C : 227 Entering Passive Mode. A1,A2,A3,A4,a1,a2
                                           C->B : PORT A1,A2,A3,A4,a1,a2
                                           B->C : 200 Okay
         C->A : STOR                       C->B : RETR
                    B->A : Connect to HOST-A, PORT-a

I.e. One server is put in passive mode and one in active mode. A good explanation of the difference can be found  here. GridFTP states support for 3rd-party transfer in the specification. In practice this means that both passive and active FTP commands are extended to support striping across multiple data ports.

On a casual read of the specifications it appears that, provided a server supports active mode, the main issue is with the client. It must support making control connections to 2 servers simultaniously and issueing the necessary PASV, PORT, STOR and RETR commands.

Unlike GridFTP, bbFTP is not an extension of RFC-959. However, it claims to support both passive and "non-passive" mode. Therefore, it may be possible to adapt the client to do 3rd party transfer along the lines shown above. This may be a significant ammount of work and any consideration of it should be balanced against the work required to make GridFTP support NDG Security.

Change History

comment:1 Changed 13 years ago by spascoe

  • Status changed from new to assigned

The issue as I see it is do we need 3rd party transfer as a seperate service?

NDG services, such as GeoSplat?, may need to bring data to themselves as part of fulfilling a request from a user. This is 3rd party transfer of a sort (involving user, GeoSplat? and 3rd service), but might be better dealt with on a per-service basis. By which I mean GeoSplat? simply invoking the DeliveryService as a client.

I think this is the common sense way of doing it.

comment:2 Changed 13 years ago by spascoe

  • Cc selatham added

I have created a wiki page for recording the issues at ThirdPartyDataTransfer. I have investigated how GridFTP does it, which has proved instructive.

comment:3 Changed 13 years ago by spascoe

  • Description modified (diff)

comment:4 Changed 13 years ago by spascoe

  • Description modified (diff)

comment:5 Changed 13 years ago by lawrence

  • Milestone changed from ALPHA to BETA

comment:6 Changed 12 years ago by spascoe

  • Status changed from assigned to closed
  • Resolution set to wontfix

This ticket is now completely out of scope for NDG2

comment:7 Changed 11 years ago by lawrence

  • Keywords Delivery added
  • Component changed from T05_Delivery to ndg2

moved to component ndg2 (obsolete) as part of ndg2 cleanup

Note: See TracTickets for help on using tickets.