File Transfer (FT) Procedure

OS agents are designed for the transfer of files. Doing so requires an agent to be installed on the source computer and on the target computer. Files are transferred in a secured and encrypted way.

As of version 9.00A, the Automation Engine provides an optimized and improved file transfer procedure. This new protocol is only used if the participating agents are of version 9.00A or later. For compatibility reasons, the old procedure is used if one of the agents has an older version.

Old File Transfer Protocol (Up To 8.00A)

Both agents require a connection for the file transfer to start. One of the work processes contacts the communication process that is connected to the source agent and informs it about the connection request. The communication process passes the information on to the agent. The agent uses the included information in order to connect to the target agent.

If the attempt to establish a connection fails due to Firewall settings, for instance), it uses the same information in order to contact the target agent. The target agent now tries to establish a connection to the source agent.

The file transfer can start as soon as the two agents are connected with each other. Status messages are regularly sent to the Automation Engine in order to track the progress. The Detail window of the file transfer task in the User Interface shows the number of bytes that have already been transferred.

As opposed to the new FT protocol, all file transfer files are sent via a connection (FT connection). Blocks of different files may be transferred alternately. The Automation Engine monitors the whole file transfer procedure and instructs the source agent to send the individual files.

The administrator can determine that the connection between the two agents should end after the files have been transferred. This is done in the UC_HOSTCHAR_DEFAULT variable using the DISCONNECT_AFTER_FT setting.

New File Transfer Protocol (as of 9.00A)

As of AE v9, the Automation Engine sends the complete file transfer request (including wildcard specifications in partially qualified file transfers) to the source agent. The sending agent is responsible for resolving the request (determining the files).

Establishing the Connection

The sending agent tries to establish a connection to the receiving agent. If this attempt fails (due to Firewall settings, for instance), it notifies the Automation Engine. The file transfer request is then sent to the receiver, which now tries to establish a connection to the sender. After the connection has been established, the receiving agent transfers the request to the sender.

Checking the Disk Space

Depending on the OS, before starting a file transfer the system checks whether there is enough disk space on the target platform and, if not, it will allocate it.

Handling File Transfers

For each file transfer, the new protocol establishes a separate connection between the agents. The files are always sent through a connection one after the other. Each file transfer is handled in a separate thread or process if this is supported by the agent. Therefore, several file transfers can be processed independently of each other.

The DISCONNECT_AFTER_FT setting of the UC_HOSTCHAR_DEFAULT variable does not affect the new protocol because the system ends the connections after the File transfers have been completed.

The agents of the following operating systems support threads:

NSK handles each file transfer with a separate process. Therefore, the NSK agent has a second port especially for file transfers.

File transfers are not affected by any connection errors between the Automation Engine and the agents. The actual status of the file transfer is sent to the Automation Engine as soon as the connection could be re-established.

The NFS "root squash" security setting causes problems in file transfers with UNIX agents if it is used in combination with the old FT protocol. The reason is that file transfers are always executed under the UNIX "root" user. This error does not occur in the new protocol because file transfers under UNIX always run under the user specified in the Login object.

The following file transfer procedures are provided in order to ensure a reliable transfer of files:

File transfer procedure Explanation
Transmission security The accuracy of transferred data is verified with an MD5 checksum verifier that is embedded in the data stream. Data is verified in packets.
Consistency check for restarted file transfers

In contrast to the old protocol, you cannot repeat individual file transfer files selectively. Erroneous file transfers can be repeated from the last restart point.

At particular intervals, the agents automatically create restart points while the files are being transferred (setting FT_RESTARTINFO_INTERVAL in the UC_HOSTCHAR_DEFAULT variable). The agent stores this information locally on its computer in StatusStore files. If an error occurs, the file transfer can be restarted from the file last restart point (Restart option: "From last restart position"). This function saves time especially if most of a huge file has already been transferred.

You can use the FT_RESTARTINFO_LIFETIME and FT_RESTARTINFO_CHECK (UC_HOSTCHAR_DEFAULT) settings in order to specify that StatusStore files should be deleted.

To ensure that the target file complies with the source file after it has been successfully restarted, the transferred data is checked against an MD5 checksum. When it creates the restart point, the system also retrieves the MD5 checksum and stores it in the StatusStore file. The checksums differ if the partially transferred file has been changed on the computer of the receiving agent, and the restart results in an error.

To save transmission time, MD5 checksums of files that are smaller than 1 MB are not calculated.

You can deactivate the MD5 checksum using the FT_USE_MD5 setting in the UC_HOSTCHAR_DEFAULT variable .

Depending on the particular agent platform, the StatusStore files are stored in the following directories:

Platform Directory File name Peculiarity
Windows The Temp directory of the agent

FTNNNNNNN.sts

where NNNNNNN is the file transfer's RunID that has been converted to a 7-letter string. You can use the ALPHA2RUNNR script element in order to convert it to the regular 10-digit RunID.

StatusStore file per file transfer
BS2000 The Temp directory of the agent FTNNNNNNN.sts StatusStore file per file transfer
Unix/VMS The Temp directory of the agent FTNNNNNNN.sts StatusStore file per file transfer
OS/400

Depending on the INI-file parameter store_type=

IFS: FTNNNNNNN.sts
QSYS: Object with the name FTNNNNNNN and the type USRSPC
StatusStore file per file transfer
NSK

Sub-volume in the configuration file (see the NSK agent installation).

The INI file of the agent:

Section [FT-STATUS-STORE]

Four StatusStore files that include all restart information
z/OS  

StatusStore dataset

See: Installing the z/OS Agent

A StatusStore dataset that includes all restart information

See also: