FileTransfer

FileTransfer

Multi-Server Operation

---

 File Transfer (FT) Procedure

OS agents are designed for the transfer of files. Doing so requires an agent to be installed on the source computer and on the target computer. Files are transferred in a secured and encrypted way.

Automation Engine version 9.00A provides an optimized and improved file transfer procedure. This new protocol is only used if the participating agents are of version 9.00A or later. For compatibility reasons, the old procedure is used if one of the agents has an older version.

Old File Transfer Protocol (Up To 8.00A)

Both agents require a connection for the file transfer to start. One of the work processes contacts the communication process that is connected to the source agent and informs it about the connection request. The communication process passes the information on to the agent. The agent uses the included information in order to connect to the target agent.

If the Automation Engine's attempt to establish a connection fails (for example, because of Firewall settings), it uses the same information in order to contact the target agent. The target agent now tries to establish a connection to the source agent.

The file transfer can start as soon as the two agents are connected with each other. Status messages are regularly sent to the Automation Engine in order to track the progress. The file transfer task's Detail Window in the UserInterface shows the number of bytes that have already been transferred.

Unlike with the new FT protocol, all file transfer files are sent via a connection (FT connection). Blocks of different files may be transferred alternately. The Automation Engine monitors the whole file transfer procedure and instructs the source agent to send the individual files.

The administrator can determine that the connection between the two agents should end after the files have been transferred. This is done in the variable UC_HOSTCHAR_DEFAULT using the setting DISCONNECT_AFTER_FT. 

New File Transfer Protocol (as of 9.00A)

As of AE v9, the Automation Engine sends the complete file transfer request (including wildcard specifications in partially qualified file transfers) to the source agent. The sending agent is responsible for resolving the request (determining the files).

Connection establishment

The sending agent tries to establish a connection to the receiving agent. If this attempt fails (for example, because of Firewall settings), it notifies the Automation Engine. The FT request is then sent to the receiver which now tries to establish a connection to the sender. After the connection has been established, the receiving agent transfers the FT request to the sender.

Checking the disk space

Depending on OS, the system will check before starting a file transfer whether there is enough disk space on the target platform and, if not, will allocate it.

• BS2000: Estimated disk space is allocated if the option Keep original file attributes has been activated in the FileTransfer object.
• NSK: Disk space is not checked.
• OS/390 - Native file system: Disk space is allocated using the attribute SPACE for the file transfer's target.
• OS/390 - USS file system: Disk space is not checked.
• OS/400 - Native file system: Disk space is allocated if either "Keep original file attributes" has been set or the attribute SIZE has been specified for the target.
• OS/400 - IFS file system: The available disk space is checked.
• UNIX: Depending on the ini-Parameter ft_check_free_disk_space=
• VMS: The available disk space is checked.
• Windows: Depending on the ini-Parameter ft_check_free_disk_space=

Handling file transfers

For each file transfer, the new protocol establishes a separate connection between the agents. The files are always sent through a connection one after the other. Each file transfer is handled in a separate thread or process if this is supported by the agent. Therefore, several file transfers can be processed independently of each other.

The setting DISCONNECT_AFTER_FT of the variable UC_HOSTCHAR_DEFAULT does not affect the new protocol because the system ends the connections after the Filetransfers have been completed.

The agents of the following operating systems support threads:

• OS/400
• Unix
• Windows
• z/OS

NSK handles each file transfer with a separate process. Therefore, the NSK agent has a second port especially for file transfers.

File transfers are not affected by any connection errors between Automation Engine and agents - they are continued. The file transfer's actual status is sent to the Automation Engine as soon as the connection could be re-established.

The NFS security setting "root squash" causes problems in file transfers with UNIX agents if it is used in combination with the old FT protocol. The reason is that file transfers are always executed under the UNIX user "root". This error does not occur in the new protocol because file transfers under UNIX always run under the user specified in the Login object.

The following file transfer procedures are provided in order to ensure a reliable transfer of files:

Transmission security

The accuracy of transferred data is verified with an MD5 checksum verifier that is embedded in the data stream. Data is verified in packets.

Consistency check for restarted file transfers

Unlike with the old protocol, you cannot repeat individual file transfer files selectively. Erroneous file transfers can be repeated from the last restart point.

At particular intervals, the agents automatically create restart points while the files are being transferred (setting FT_RESTARTINFO_INTERVAL in the variable UC_HOSTCHAR_DEFAULT). The agent stores this information locally on its computer in StatusStore files. If an error occurs, the file transfer can be restarted from the file's last restart point (Restart option: "From last restart position"). This function saves time especially if most of a huge file has already been transferred.

You can use the settings FT_RESTARTINFO_LIFETIME and FT_RESTARTINFO_CHECK (UC_HOSTCHAR_DEFAULT) in order to specify that StatusStore files should be deleted.

In order to ensure that the target file complies with the source file after it has successfully been restarted, the transferred data is checked against an MD5 checksum.When it creates the restart point, the system also retrieves the MD5 checksum and stores it in the StatusStore file. The checksums differ if the partially transferred file has been changed on the receiving agent's computer, and the restart results in an error.

To save transmission time, MD5 checksums of files that are smaller than 1 MB are not calculated.

You can deactivate the MD5 checksum using the setting FT_USE_MD5 in the variable UC_HOSTCHAR_DEFAULT.

Depending on the particular agent platform, the StatusStore files are stored in the following directories: 

See also:

FileTransfer
FileTransfer - Execution
Multi-Server Operation
Establishing a Connection