Setting S3 Job Properties Through Scripts

Attributes defined for an object affect how the object is processed. Script statements allow you to dynamically change those object attributes and assign new values when the task is generated. The new values are only valid for the current generation and are not permanently stored in the object.

Note:

The Failed Operation Parameters and the Server-Side Encryption Parameters apply partly or completely to most of the S3 Jobs. They are listed only once to avoid redundancies.

You can define the values of any of the settings of all S3 Jobs using scripts. You do this by using :PUT_ATT commands on the Pre-Process object page.

For more information, see the Automic Automation product documentation:

This page includes the following:

S3 Copy File Job Attributes

The attributes available for S3 Copy File Jobs are the following:

  • connection

    Sets the S3 Connection object.

    Value: text

  • region

    Sets the relevant Region.

    Defining the region in the job is optional and only relevant for AWS. If you choose not to define it, the job takes the URL defined in the Connection object.

    However, if you decide to define the region in the job, make sure that the definition matches the one defined in the Connection object that you have selected for the job. If both region definitions do not match, the job execution fails and an error message is logged in both, the Agent (PLOG) and the Job (REP) reports (see Monitoring S3 Jobs).

    Value: text

  • sourcebucket

    Sets the name of the source bucket that you want to use.

    Value: text

  • useregex

    Defines if you want to use a regex or not.

    Values: 0 (do not use regex), 1 (use regex)

  • sourcefilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the source file name or the path of where the source file is located.

    Value: text

  • source

    This option is available only if you have selected the Use Regex option.

    Sets where the relevant files are located in the bucket.

    Value: text

  • destinationbucket

    Sets the destination bucket name that you want to use.

    Value: text

  • destinationfilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the destination file name or the path to where the file should be copied in the bucket.

    Value: text

  • destinationpath

    This option is available only if you have selected the Use Regex option.

    Sets the path to the destination to which you want to move the files.

    Warning!

    Make sure you define a folder and not a single file (path). Otherwise, you will overwrite the contents of the folder with a single file.

    Value: text

  • queryparam

    Define query parameters to customize your request.

    There is no restriction on the parameters that you can use in a query. All URI request parameters for all S3 Jobs are supported. For more information, refer to the official AWS S3 and GCS documentations, respectively.

    Value: text

Additionally, you can also set Failed Operation Parameters (retry and check for overwrites) and, if using AWS S3, Server-Side Encryption Parameters for this Job, see S3 Failed Operation Parameters and AWS S3 Server-Side Encryption Parameters.

S3 Delete File Job Attributes

The attributes available for S3 Delete File Jobs are the following:

  • connection

    Sets the S3 Connection object.

    Value: text

  • region

    Sets the relevant Region.

    Defining the region in the job is optional and only relevant for AWS. If you choose not to define it, the job takes the URL defined in the Connection object.

    However, if you decide to define the region in the job, make sure that the definition matches the one defined in the Connection object that you have selected for the job. If both region definitions do not match, the job execution fails and an error message is logged in both, the Agent (PLOG) and the Job (REP) reports (see Monitoring S3 Jobs).

    Value: text

  • bucketname

    Sets the name of the bucket that you want to use.

    Value: text

  • useregex

    Defines if you want to use a regex or not.

    Values: 0 (do not use regex), 1 (use regex)

  • filename

    This option is available only if you have not selected the Use Regex option.

    Sets the name of the relevant file.

    Value: text

  • filepattern

    This option is available only if you have selected the Use Regex option.

    Enter the regular expression that define the file pattern matching the files that you want to delete.

  • queryparam

    Define query parameters to customize your request.

    There is no restriction on the parameters that you can use in a query. All URI request parameters for all S3 Jobs are supported. For more information, refer to the official AWS S3 and GCS documentations, respectively.

    Value: text

Additionally, you can also set if you want to retry Failed Operation Parameters, see S3 Failed Operation Parameters.

S3 Download File Job Attributes

The attributes available for S3 Download File Jobs are the following:

  • connection

    Sets the S3 Connection object.

    Value: text

  • region

    Sets the relevant Region.

    Defining the region in the job is optional and only relevant for AWS. If you choose not to define it, the job takes the URL defined in the Connection object.

    However, if you decide to define the region in the job, make sure that the definition matches the one defined in the Connection object that you have selected for the job. If both region definitions do not match, the job execution fails and an error message is logged in both, the Agent (PLOG) and the Job (REP) reports (see Monitoring S3 Jobs).

    Value: text

  • sourceBucket

    Sets the name of the source bucket that you want to use.

    Value: text

  • useregex

    Defines if you want to use a regex or not.

    Values: 0 (do not use regex), 1 (use regex)

  • sourcefilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the source file name or the path of where the source file is located.

    Value: text

  • source

    This option is available only if you have selected the Use Regex option.

    Sets where the relevant files are located in the bucket.

    Value: text

  • destinationfilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the destination file name or the path to where the file should be downloaded in the bucket.

    Value: text

  • destinationpath

    This option is available only if you have selected the Use Regex option.

    Sets the path to the destination to which you want to move the files.

    Warning!

    Make sure you define a folder and not a single file (path). Otherwise, you will overwrite the contents of the folder with a single file.

    Value: text

  • queryparam

    Define query parameters to customize your request.

    There is no restriction on the parameters that you can use in a query. All URI request parameters for all S3 Jobs are supported. For more information, refer to the official AWS S3 and GCS documentations, respectively.

    Value: text

You can also define file permissions for downloaded files using the following attributes:

  • filePermission

    Sets the file permission definition.

    Value: text

  • fileOwner

    Sets the owner of the downloaded file who will have the specified permissions.

    Value: text

  • fileOwnerGroup

    Unix only. Sets the owner group as specified on your Unix environment. This group will have the specified permissions.

    Value: text

  • onInvalidPermissionSettings

    Allows you to define if the job should fail or succeed when the permission settings are either left empty or incorrect.

    • Fail (default)

      Select this option if the job should fail when the permission settings are either left empty or incorrect.

    • Succeed

      Select this option if the job should succeed when the permission settings are either left empty or incorrect.

Additionally, you can also retry Failed Operation Parameters and, if using AWS S3, Server-Side Encryption Parameters for this Job, see S3 Failed Operation Parameters and AWS S3 Server-Side Encryption Parameters.

S3 Exist File Job Attributes

The attributes available for S3 Exist File Jobs are the following:

  • connection

    Sets the S3 Connection object.

    Value: text

  • region

    Sets the relevant Region.

    Defining the region in the job is optional and only relevant for AWS. If you choose not to define it, the job takes the URL defined in the Connection object.

    However, if you decide to define the region in the job, make sure that the definition matches the one defined in the Connection object that you have selected for the job. If both region definitions do not match, the job execution fails and an error message is logged in both, the Agent (PLOG) and the Job (REP) reports (see Monitoring S3 Jobs).

    Value: text

  • bucketname

    Sets the name of the bucket that you want to use.

    Value: text

  • filename

    Sets the name of the relevant file.

    Value: text

  • useregex

    Defines if you want to use a regex or not.

    Values: 0 (do not use regex), 1 (use regex)

  • queryparam

    Allows you to filter the query and therefore the query response.

    There is no restriction on the parameters that you can use in a query. All URI request parameters for all S3 Jobs are supported. For more information, refer to the official AWS S3 and GCS documentations, respectively.

    Value: text

Additionally, if you use AWS S3 and the file that you want to check was uploaded to the bucket using a custom encryption type (SSE-C), you need to provide an algorithm and a key to be able to check its existence, see AWS S3 Server-Side Encryption Parameters.

S3 Monitor File Job Attributes

The attributes available for S3 Monitor File Jobs are the following:

  • connection

    Sets the S3 Connection object.

    Value: text

  • monitorjobtype

    Sets the type of job that you want to monitor in the specified bucket.

    Value: text

  • region

    Sets the relevant Region.

    Defining the region in the job is optional and only relevant for AWS. If you choose not to define it, the job takes the URL defined in the Connection object.

    However, if you decide to define the region in the job, make sure that the definition matches the one defined in the Connection object that you have selected for the job. If both region definitions do not match, the job execution fails and an error message is logged in both, the Agent (PLOG) and the Job (REP) reports (see Monitoring S3 Jobs).

    Value: text

  • bucketname

    Sets the name of the bucket that you want to use.

    Value: text

  • filename

    Sets the name of the relevant file.

    Value: text

  • sortingStrategy

    Sets the sorting strategy that you want to use when using a regex.

    Values: native_order, alpha_asc, alpha_desc, time_asc, time_desc, size_asc, size_desc

  • useregex

    Defines if you want to use a regex or not.

    Values: 0 (do not use regex), 1 (use regex)

  • steadystate

    Sets the amount of time in seconds (s), minutes (m), or hours (h) the file should remain in a steady state before being ready for further use.

    Value: text

  • sleepinterval

    Sets the amount of time in seconds before resubmitting the monitor status request.

    Value: text

  • queryparam

    Allows you to filter the query and therefore the query response.

    There is no restriction on the parameters that you can use in a query. All URI request parameters for all S3 Jobs are supported. For more information, refer to the official AWS S3 and GCS documentations, respectively.

    Value: text

  • returnfileuri

    Sets if you want the execution details to include the full URI of the file name.

    Values: 0 (do not return file URI), 1 (return file URI)

Additionally, if you use AWS S3 and the file that you want to monitor was uploaded to the bucket using a custom encryption type (SSE-C), you need to provide an algorithm and a key to be able to monitor it, see AWS S3 Server-Side Encryption Parameters.

S3 Move File Job Attributes

The attributes available for S3 Move File Jobs are the following:

  • connection

    Sets the S3 Connection object.

    Value: text

  • region

    Sets the relevant Region.

    Defining the region in the job is optional and only relevant for AWS. If you choose not to define it, the job takes the URL defined in the Connection object.

    However, if you decide to define the region in the job, make sure that the definition matches the one defined in the Connection object that you have selected for the job. If both region definitions do not match, the job execution fails and an error message is logged in both, the Agent (PLOG) and the Job (REP) reports (see Monitoring S3 Jobs).

    Value: text

  • sourcebucket

    Sets the name of the source bucket that you want to use.

    Value: text

  • useregex

    Defines if you want to use a regex or not.

    Values: 0 (do not use regex), 1 (use regex)

  • sourcefilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the source file name or the path of where the source file is located.

    Value: text

  • source

    This option is available only if you have selected the Use Regex option.

    Sets where the relevant files are located in the bucket.

    Value: text

  • destinationbucket

    Sets the destination bucket name that you want to use.

    Value: text

  • destinationfilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the destination file name or the path to where the file should be moved in the bucket.

    Value: text

  • destinationpath

    This option is available only if you have selected the Use Regex option.

    Sets the path to the destination to which you want to move the files.

    Warning!

    Make sure you define a folder and not a single file (path). Otherwise, you will overwrite the contents of the folder with a single file.

    Value: text

  • queryparam

    Define query parameters to customize your request.

    There is no restriction on the parameters that you can use in a query. All URI request parameters for all S3 Jobs are supported. For more information, refer to the official AWS S3 and GCS documentations, respectively.

    Value: text

Additionally, you can also set Failed Operation Parameters (retry and check for overwrites) and, if using AWS S3, Server-Side Encryption Parameters for this Job, see S3 Failed Operation Parameters and AWS S3 Server-Side Encryption Parameters.

S3 Upload File Job Attributes

The attributes available for S3 Upload File Jobs are the following:

  • connection

    Sets the S3 Connection object.

    Value: text

  • region

    Sets the relevant Region.

    Defining the region in the job is optional and only relevant for AWS. If you choose not to define it, the job takes the URL defined in the Connection object.

    However, if you decide to define the region in the job, make sure that the definition matches the one defined in the Connection object that you have selected for the job. If both region definitions do not match, the job execution fails and an error message is logged in both, the Agent (PLOG) and the Job (REP) reports (see Monitoring S3 Jobs).

    Value: text

  • useregex

    Defines if you want to use a regex or not.

    Values: 0 (do not use regex), 1 (use regex)

  • sourcefilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the source file name or the path of where the source file is located.

    Value: text

  • source

    This option is available only if you have selected the Use Regex option.

    Sets where the relevant files are located in the bucket.

    Value: text

  • bucketname

    Sets the name of the bucket that you want to use.

    Value: text

  • destinationfilepath

    This option is available only if you have not selected the Use Regex option.

    Sets the destination file name or the path to where the file should be uploaded in the bucket.

    Value: text

  • destinationpath

    This option is available only if you have selected the Use Regex option.

    Sets the path to the destination to which you want to move the files.

    Warning!

    Make sure you define a folder and not a single file (path). Otherwise, you will overwrite the contents of the folder with a single file.

    Value: text

  • queryparam

    Define query parameters to customize your request.

    There is no restriction on the parameters that you can use in a query. All URI request parameters for all S3 Jobs are supported. For more information, refer to the official AWS S3 and GCS documentations, respectively.

    Value: text

Additionally, you can also set Failed Operation Parameters (retry and check for overwrites) and, if using AWS S3, Server-Side Encryption Parameters for this Job, see S3 Failed Operation Parameters and AWS S3 Server-Side Encryption Parameters.

S3 Failed Operation Parameters

The Failed Operation Parameters section allows you to define if you want to try to execute failed operations again or not.

These are the attributes available to define how to deal with failed operations:

  • retry

    Sets if you want to retry to execute failed operations.

    Values: 0 (do not retry), 1 (retry)

  • retrycount

    Define the number of retry attempts.

    Value: text

  • retrydelay

    Define the time in seconds between attempts.

    Value: text

  • overwrite

    Defines if you want the system to check for overwrites or not.

    Values: 0 (do not check for overwrites), 1 (check for overwrites)

  • overwriteaction

    Allows you to check if the destination bucket already contains the files that you want to put in it and, if so, allows you to choose an action.

    Values:

    • Overwrite existing files (default)

      Set this option if you want to overwrite the existing files on the destination.

    • Fail the job and skip the operation

      Set this option if you want to skip the operation and fail the job instead of overwriting files on the destination.

AWS S3 Server-Side Encryption Parameters

The attributes available to set AWS S3 server-side encryption are the following:

  • serverSideEncryptionSetting

    Defines if you want to specify and encryption key.

    Values: 0 (do not specify an encryption key), 1 (specify an encryption key)

  • encryptionSetting

    Defines if you want to override default bucket encryption.

    Values: 0 (do not override default bucket encryption), 1 (override default bucket encryption)

  • encryptionType

    Sets the encryption type selected.

    Values:

    • SSE-3 (server-side encryption with AWS S3 managed keys)

    • SSE-KMS (server-side encryption with AWS Key Management Service keys)

    • DSSE-KMS (dual-layer server-side encryption with AWS Key Management Service keys)

    • SSE-C (custom encryption key)

    The SSE-3 encryption type does not require further definition. When using the encryption types SSE-KMS or DSSE-KMS, define the following:

    • encryptionARN

      Sets the ARN relevant for the encryption.

      Value: text

    When using the encryption type SSE-C, define the following:

    • encryptionCustomerAlgo

      Sets the algorithm used to upload a file. Only AES256 is supported.

      Value: text

    • encryptionCustomerKey

      Sets the encryption key that you want to use to execute the job.

      Value: text

    For Copy and Move File Jobs, if the file was originally uploaded to the bucket using custom encryption, you also need to define the following:

    • encryptionSourceCustomerAlgo

      Sets the customer algorithm that was used to upload the file. Only AES256 is supported.

      Value: text

    • encryptionSourceCustomerKey

      Sets the customer key that was used to upload the file.

      Value: text

See also: