wiki:DataTransferMethod

Version 1 (modified by joe, 19 years ago) (diff)

--

Data Transfer Methods

The VSO provides a number of keywords to describe the methods in which data products are made available from each of the VSO Data Providers.

Some transfer methods may require additional information to complete the request.

For the most recent list of Transfer Method Keywords, please see

http://vso.nascom.nasa.gov/cgi-bin/show_details.pl?keyword=METHOD_PREFIX

Each basic transfer type will typically be modified by one or more sub-types to better describe the packaging of the data products.

The most recent list of subtypes can be found at

http://vso.nascom.nasa.gov/cgi-bin/show_details.pl?keyword=METHOD_SUFFIX

A listing of which methods each Data Provider supports is available at

http://vso.nascom.nasa.gov/cgi-bin/show_details.pl?keyword=PROVIDER

Please note that this is not a complete list of available transfer methods, and that as of this writing, at least one of them isn't currently being used by any Data Providers. The list of transfer methods used by VSO Data Providers may change as new Data Providers are added, or demand for other transfer types are identified

Basic Data Transfer Methods

URL

There is a URL that can be followed to get to the data of interest.

STAGING

The data must be processed by the Data Provider before it is made available for download. Typically, for STAGING transfers, the Data Provider will request an e-mail address that they can send notification to when the process is complete.

STAGING responses should contain Detail elements that give an indication as to how long the request will take to complete.

OFFLINE

The data can be sent encoded on physical media, and sent through parcel post. Typically, for OFFLINE transfers, the Data Provider will request an e-mail address or phone number to coordinate the process. The Data Provider may request information for shipping the products, such as the recipient's name and shipping address.

OFFLINE responses should contain Detail elements that give an indication as to how long it will take for the Data Provider to respond to the request, to work out necessary details, such as the possibilility of cost-recovery.

PUSH

The data will transfer the data to you, if you give them the appropriate connection information for them to do so. Typically, for PUSH transfers, the Data Provider will request an e-mail address that they can send notification to when the process is complete, or to work out any difficulties that may occur. They will also require connection information so they can transfer the data to the requesting user.

PUSH responses should contain Detail elements that give an indication as to how long the request may take to process. The time is typically an indicaton as to when the process may start, as the Data Provider would have no knowledge of the transfer rate to the user's system as this time.

Please check with your local information or network security office before coordinating a PUSH of data, as it will often times be blocked a border router or firewall, or it may fall under specific security policies within your organization.

Transfer Method Subtypes

A Data Provider may support multiple methods, of one or more basic types. Specific methods are specified by giving a basic type, a hypen, and a sub-type.

Subtypes vary by basic types, but for the most part, specify the method of encoding the data:

URL

The sub-type for URL specifies the compression and packaging of the data. For example,

  • URL-FILE A single file available for each URL.
  • URL-packaged A special case, requesting few URLs to download all of the requested files. (ie, URL, but not URL-FILE)
  • URL-TAR Each URL points to a tar'd group of files.
  • URL-TAR_GZ Each URL points to a tar'd group of files, which has been gzip'd.
  • URL-ZIP Each URL points to a group of files in a ZIP archive.

STAGING

The sub-type for STAGING specifies the compression and packaging of the data. For example,

  • STAGING-TAR The files are archived using the unix tar command or its equivalent.
  • STAGING-TAR_GZ The files are archived using the unix tar command, and then compressed using gzip.
  • STAGING-GZ_TAR The individual files are compressed using the gzip command, and then archived into a tarball.
  • STAGING-ZIP The files are archived using ZIP compression.

OFFLINE

The sub-type for OFFLINE specifies the type of media that the Data Products should be written to for shipment. For example,

  • OFFLINE-CD The data will be written out to a Compact Disc.
  • OFFLINE-DAT The data will be written out to a 4mm Digital Audio Tape. (yes, I know, it's technically not a DAT when it's not holding audio information, but it's what people are used to calling it)
  • OFFLINE-EXABYTE8200 The data will be written out to an Exabyte model 8200 tape.
  • OFFLINE-EXABYTE8500 The data will be written out to an Exabyte model 8500 tape.

PUSH

The sub-type for PUSH specifies the protocol that will be used to transfer the data. For example,

  • PUSH-RCP The data will be transfered using the Remote Copy Protocol. This will typically require the requesting user to configure an appropriate rhosts file, and the Data Provider should specify their server's name in the details element of their response.
  • PUSH-FTP The data will be transfered using the File Transfer Protocol. Please note that many places may prefer anonymous FTP, where no password is used, to prevent it from being sniffed while in transit, while others prefer that anonymous FTP not be used due to the potential for unauthorized parties to gain access to place files on the system.
  • PUSH-SCP The data will be transfered using the Secure Copy Protocol.
  • PUSH-SFTP1 The data will be transfered using the Secure File Transfer Protocol, version 1. Please note that a system that supports SFTP2 may not also support SFTP1, although it is designed to degrade gracefully should SFTP1 also be installed.
  • PUSH-SFTP2 The data will be transfered using the Secure File Transfer Protocol, version 2. Please note that a system that supports SFTP2 may not also support SFTP1, although it is designed to degrade gracefully should SFTP1 also be installed.