GT 3.9.3 Development Release Notes for GridFTP

Component Overview

GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.

Feature Summary

Features new in release Globus Toolkit 3.9.3

  • A new, complete reimplementation of the server.
  • Support for striping (using multiple hosts to move a single file).
  • This new implementation will greatly ease new feature additions and modifications of the server (new commands, new data sources such as mass storage devices, etc.), maintainability, and resolves a licensing issue that was discovered.

Features that continue to be supported from previous versions

  • GSI security: This is PKI based defacto standard security system used in Grid applications. Kerberos is also possible, but not supported and can be difficult due to divergence in the capabilities of GSI and Kerberos.
  • Third-party transfers: Very common in Grid applications, this is where a client mediates a transfer between two servers (both likely at remote sites) rather than between the server and itself (called a client/server transfer).
  • Partial file access: Regions of a file may be accessed by specifying an offset into the file and the length of the block desired.
  • Reliability/restart: The receiving server periodically (the default is 5 seconds, but this can be changed) sends “restart markers” to the client. This marker is a messages specifying what bytes have been successfully written to the disk. If the transfer fails, the client may restart the transfer, and provide these markers (or an aggregated equivalent marker) and the transfer will pick up where it left off. This can include “holes” in the file.
  • Large file support: All file sizes, lengths, and offsets are 64 bits in length.
  • Data channel reuse: Data channel can be held open and reused if the next transfer has the same source, destination, and credentials. This saves the time of connection establishment, authentication, and delegation. This can be a huge performance difference when moving lots of small files.
  • Integrated instrumentation (Performance Markers)
  • Logging/audit trail (Extensive Logging in the server)
  • Parallel transfers (Multiple TCP streams between a pair of hosts)
  • TCP Buffer size control (Protocol supports Manual and Automatic; Only Manual Implemented)
  • Server-side computation (Extended Retrieve (ERET) / Extended Store (ESTO) commands)
  • Based on Standards: RFC 959, RFC 2228, RFC 2389, IETF Draft MLST-16 , GGF GFD.020

Other Supported Features

  • On the client side, we provide a scriptable client side tool called globus-url-copy. This tool can take advantage of all the GridFTP protocol features and can also do protocol translation between FTP, HTTP, HTTPS, and posix file IO on the client machine.
  • We also provide a set of development libraries and APIs for developers wishing to add GridFTP functionality to their application.

Deprecated Features

  • None

Bug Fixes

  • 1883 globus_ftp_contol server does not decode ADAT properly.
  • 1928 Strange problems: Busy wait and address in use.
  • 2036 New gridftp, globus-gridftp-server, does not use globus_gss_assist_map_and_authorize
  • 2128 gridftp server checks ownership of hostcert after setuid
  • 2129 Gridftp logging levels (is 7>10?)
  • 2132 Server responds with "Valid credentials could not be found..."

Known Problems

  • GridFTP Server
    • There are some small memory leaks, though they should not grow much.
    • Threaded builds are not expected to work properly.
    • Error resonses are unclear and inconsistent.

Technology Dependencies

GridFTP depends on the following GT components:

  • Pre-WS Authentication / Authorization
  • C Common Libraries
  • XIO

GridFTP depends on the following 3rd party software:

  • OpenSSL (version included in release)

Tested Platforms

Tested platforms for GridFTP

  • i386 Linux
  • ia64 Linux (TeraGrid)

We have not tested it elsewhere, but if you do and have problems, we would like to hear about it.

Backward Compatibility Summary

Protocol changes since GT version 3.2

  • None

API changes since GT version 3.2

  • None

Exception changes since GT version 3.2

  • Not Applicable (GridFTP is not Java-based)

Schema changes since GT version 3.2

  • Not Applicable (GridFTP is not SOAP-based)

For More Information

Click here for more information about this component.