Integrating Amazon MSK with Orkes Conductor

This developer guide includes the steps to integrate Amazon MSK with Orkes Conductor. This integration lets you connect the Amazon MSK (Kafka) cluster to Conductor to publish and receive messages from queues/topics.

note

Amazon MSK configuration is deprecated. Please use Apache Kafka for all future configurations.

Get Configuration Credentials from Amazon MSK

Before beginning the integration process in Orkes Conductor, you must get specific credentials, such as the Bootstrap server.

To get the Bootstrap server:

1. Login to Amazon MSK console.
2. Once logged in, the table lists all the clusters under this account for the current region.
3. Choose the cluster and click View client information on the cluster summary page. This will give you the bootstrap broker and the Apache ZooKeeper connection string.

Refer to the official documentation for more details.

Integrating with Amazon MSK as a Message Broker

Now, you have the required configuration credentials from Amazon MSK. Let’s integrate with Orkes Conductor.

1. Navigate to Integrations from the left menu on your Orkes Conductor cluster.
2. Click + New integration from the top-right corner.
3. Under the Message Broker section, choose Amazon MSK.
4. Click +Add and provide the following parameters:

Paremeters	Description
Integration Name	A name to identify the integration.
Bootstrap Server	The bootstrap server for the Amazon MSK cluster. For instructions on how to get the bootstrap server, refer to the previous section.
Sending Protocol	Choose the sending protocol for the integration. Currently supports String.
Connection Security	Choose the connection security. Currently supports SASL_SSL/SCRAM-SHA-512.
Username	The username of the Amazon MSK account.
Password	The password of the Amazon MSK account.
Consumer Group ID	Enter the Consumer Group ID from Kafka. This unique identifier helps manage message processing, load balancing, and fault tolerance within consumer groups.
Description	A description of the integration.

5. You can toggle on the Active button to activate the integration instantly.
6. Click Save.

Creating Event Handlers in Orkes Conductor

The integration is created successfully now. The next step is to create an event handler in Orkes Conductor.

To do this:

1. Navigate to Definitions > Event Handler from the left menu on your Orkes Conductor cluster.
2. Click the + Define event handler option from the top-right corner.
3. Create an event handler with the following configurations:

Event Handler Parameters	Description
Name	A name to identify the event handler definition.
Event	The event integration you have created in the following format: Type : Config Name : Topic Name Example: kafka_msk:john-test:topic_0 Note: The drop-down automatically lists the integrations you’ve added to the Conductor cluster. You can choose that and add the topic name you want to publish/receive messages.
Condition	The ECMAScript to control the message processing if required. Check out the event handler documentation for more details.
Actions	Choose the required actions to be carried out on receiving the events from MSK Kafka. It can take the following values: Complete Task Terminate Workflow Update Variables Fail Task Start Workflow Each type of action requires and supports a certain set of input parameters. Check out the event handler documentation for more details.
Active	Set this to true or false. It determines if the event handler is running or not.

A sample JSON for the event handler is as follows:

{
 "name": "kafka_msk_event_handler",
 "event": "kafka_msk:kafka_msk_test_string:events-test-1",
 "condition": "true",
 "actions": [
   {
     "action": "start_workflow",
     "start_workflow": {
       "name": "event-handler-test",
       "version": 1,
       "correlationId": "",
       "input": {
         "payload": "${$}"
       }
     },
     "expandInlineJSON": false
   }
 ],
 "active": true,
 "evaluatorType": "javascript"
}

RBAC - Governance on who can use Integrations

Once the integration is added, the next step is determining who can access these integrations.

The permissions can be granted to applications/groups within the Orkes Conductor cluster.

To provide explicit permission to Groups:

1. From the left menu on your Orkes Conductor cluster, navigate to Access Control > Groups.
2. Create a new group or choose an existing group.
3. Under the Permissions section, click +Add Permission.
4. From the Integrations sub-tab, choose the integration with required permissions.
5. Click Add Permissions. This ensures that all the group members can access these integration models in their workflows.

Similarly, you can also provide permissions to applications.

Creating Workflow in Orkes Conductor

This step involves creating a workflow with an event task in Orkes Conductor. Here, we are utilizing the MSK Kafka queue as a sink for the event.

You can quickly build a workflow from UI in Orkes Conductor.

For this,

1. Navigate to Definitions > Workflow, and click the + Define Workflow button.
2. Create a workflow with an event task with the Sink in the format kafka_msk:John-Test:topic_0, where “John-Test” is the integration name and “topic_0” is the topic to which the Conductor should send/receive messages.

Executing Workflow in Orkes Conductor

The workflow can be run using different methods. You can use the Run Workflow button for quick testing, as shown in the image below:

Upon successful execution, you can verify the message's delivery through the Amazon MSK portal.

Here, the action added in the event handler definition was to start the workflow “event-handler-test”. You can verify this from the Executions > Workflow page.