Implementation Overview for AWS EMR

There are different components required to integrate your Automation Engine and AWS EMR so that you can trigger and monitor EMR Add Steps to Cluster Jobs and Start Notebook Execution jobs from your Automic Automation system.

As a system administrator, you carry out the steps required to integrate Automic Automation and AWS EMR.

These are the steps you have to follow for the integration:

Download the AWS EMR Agent Installation Package

You can download the relevant Agent installation package from our download center at https://docs.automic.com/documentation or our marketplace at https://marketplace.automic.com/.

For more information, see Downloads and Marketplace.

Install the AWS EMR Agent

After downloading the relevant installation package, you have to install the Agent so that you can establish communication between the two systems.

Important!

Before installing the Agent, check the version compatibility information at thecompatibility matrix. For information about how to use the compatibility matrix, see the Compatibility Information.

You can install the Agent manually or in a container. The containerized Agent installation refers to building and running a Docker image hosting Java Agents in a container.

Regardless of how you install the Agents, you can use them in on-premises Automic Automation systems as well as in an Automic Automation Kubernetes Edition system.

For more information, see the Automic Automation documentation:

Installing Containerized Java Agents in the Automic Automation documentation
Agents (HOST) in the Automic Automation documentation

For more information on how to install the AWS EMR agent, see Installing the Agent for AWS EMR.

Define the AWS EMR Agent

Once you have installed the Agent, you have to configure its INI file to cover your system's requirements, for example, the connection between the Agent and the Automation Engine instance. Then, you define the Agent object in AWI. There are definition pages that are common to all Agents and some that are Agent-specific.

The Agent definition allows you to configure the communication between the Automation Engine and the target system on which your EMR jobs run. The Agent starts the execution of these jobs and makes both their monitoring and the corresponding reporting possible.

For more information, see Defining the AWS EMR Agent.

Establish the Connection between the Automation Engine and the AWS EMR Agent

Now you have to make sure that the login data required to establish the connection to the AWS EMR system is available, as otherwise, you cannot create and execute EMR jobs without leaving the Automic Web Interface.

To do so, you have to create and define AWS EMR Connection objects.

For more information, see Establishing the Connection to AWS EMR.

Define the AWS EMR Job

Now that the AWS EMR Agent is installed and defined and both systems can communicate with each other, you have to create and define the EMR jobs so that you can execute the relevant processing steps in the target system.

Once the jobs are defined, you can execute them and monitor their progress from the Automic Web Interface.

For more information, see Defining AWS EMR Job Properties.