Integrating Amazon MSK with Orkes Conductor

This developer guide includes the steps to integrate Amazon MSK with Orkes Conductor. This integration lets you connect your Amazon MSK (Kafka) cluster to Conductor to publish and receive messages from queues/topics.

Get Configuration Credentials from Amazon MSK

Before beginning the integration process in Orkes Conductor, you must get specific credentials, such as the Bootstrap server.

To get the Bootstrap server:

1. Open your Amazon MSK console.
2. Once logged in, the table lists all the clusters under this account for the current region.
3. Choose your cluster, and click View client information on the cluster summary page. This gives the bootstrap broker and the Apache ZooKeeper connection string.

Refer to the official documentation for more details.

Integrating with Amazon MSK as a Message Broker

Now, you have the required configuration credentials from Amazon MSK. Let’s integrate with Orkes Conductor.

1. Navigate to Integrations from the left menu on the Conductor cluster.
2. Click + New integration from the top-right corner of your window.
3. Under the Message Broker section, choose Amazon MSK.
4. Click +Add and provide the following parameters:

Paremeters	Description
Integration Name	A name to identify your integration.
Bootstrap Server	Provide the bootstrap server for the Amazon MSK cluster. Refer to the previous section on how to get the bootstrap server.
Sending Protocol	Choose the sending protocol for the integration. Currently supports String.
Connection Security	Choose the connection security. Currently supports SASL_SSL/SCRAM-SHA-512.
Username	Provide the username of your Amazon MSK account.
Password	Provide the password of your Amazon MSK account.
Description	Provide a description of the integration.

5. You can toggle on the Active button to activate the integration instantly.
6. Click Save.

Creating Event Handlers in Orkes Conductor

The integration is created successfully now. The next step is to create an event handler in Orkes Conductor.

To do this:

1. Navigate to Definitions > Event Handler from the left menu on your Conductor console.
2. Click the + Define event handler option from the top-right corner of the window.
3. Create an event handler with the following configurations:

Event Handler Parameters	Description
Name	Provide a name to identify your event handler definition.
Event	Provide the event integration you have created in the following format: Type : Config Name : Topic Name Example: kafka_msk:john-test:topic_0 Notes: The drop-down automatically lists the integration you’ve added to the Conductor cluster. You can choose that and add the topic name you want to publish/receive messages.
Condition	Provide the ECMAScript to control the message processing if required. Check out the event handler documentation for more details.
Actions	Choose the required actions to be carried out on receiving the events from MSK Kafka. It can take the following values: Complete Task Terminate Workflow Update Variables Fail Task Start Workflow Each type of action requires and supports a certain set of input parameters. Check out the event handler documentation for more details.
Active	Set this to true or false. It determines if the event handler is running or not.

A sample JSON for the event handler is as follows:

{
 "name": "kafka_msk_event_handler",
 "event": "kafka_msk:kafka_msk_test_string:events-test-1",
 "condition": "true",
 "actions": [
   {
     "action": "start_workflow",
     "start_workflow": {
       "name": "event-handler-test",
       "version": 1,
       "correlationId": "",
       "input": {
         "payload": "${$}"
       }
     },
     "expandInlineJSON": false
   }
 ],
 "active": true,
 "evaluatorType": "javascript"
}

RBAC - Governance on who can use Integrations

Once the integration is added, the next step is determining who can access these integrations.

The permissions can be granted to applications/groups within the Orkes Conductor cluster.

To provide explicit permission to Groups:

1. From the left menu on your Conductor cluster, navigate to Access Control > Groups.
2. Create a new group or choose an existing group.
3. Under the Permissions section, click +Add Permission.
4. From the Integrations sub-tab, choose the integration with required permissions.
5. Click Add Permissions. This ensures that all the group members can access these integration models in their workflows.

Similarly, you can also provide permissions to applications.

Creating Workflow in Orkes Conductor

This step involves creating a workflow with an event task in Orkes Conductor. Here, we are utilizing the MSK Kafka queue as a sink for the event.

You can quickly build a workflow from UI in Orkes Conductor.

For this,

1. Navigate to Definitions > Workflow, and click the Define Workflow button.
2. Create a workflow with an event task with the Sink in the format kafka_msk:John-Test:topic_0, where “John-Test” is the integration name and “topic_0” is the topic to which the Conductor should send/receive messages.

Executing Workflow in Orkes Conductor

The workflow can be run using different methods. You can use the Run Workflow button for quick testing, as shown in the image below:

Upon successful execution, you can verify the message's delivery through the Amazon MSK portal.

Here, the action added in the event handler definition was to start the workflow “event-handler-test”. You can verify the same from the Executions > Workflow page.