Skip to main content

Event Streaming with Orkes Conductor & SQS Queues

This developer guide includes the steps to integrate Orkes Conductor with Amazon SQS (Simple Queue Service). The integration involves updating the access policy for the SQS queue & creating a workflow in Orkes Conductor for receiving events from the SQS messaging system.

Steps to Integrate Conductor Cluster with External-hosted SQS

Prerequisites:

  • AWS hosted Conductor cluster
  • Access to AWS IAM to manage permissions

Updating Access Policy for SQS Queue in AWS Cluster

  1. Reach out to the Orkes team to identify the IAM roles associated with the Conductor worker and server pods.
  2. Navigate to the AWS console and locate the SQS queue.
  3. Update the access policy to grant permissions for specific actions (e.g., ReceiveMessages, DeleteMessage, SendMessage, ChangeMessageVisibility, GetQueueAttributes, SetQueueAttributes, GetQueueUrl) to the identified IAM role.

Access Policy in AWS Web Console

Here’s a sample policy snippet:

{
"Sid": "test",
"Effect": "Allow",
"Principal": {
<IAM-from-step-1>
},
"Action": "SQS:*",
"Resource": "arn:aws:sqs:us-west-2:your-aws-account-id:queue-name"
}

Creating Workflow in Orkes Conductor

This step involves creating a workflow with an event task in Orkes Conductor. Here, we are utilizing an SQS queue as a sink for the event.

You can quickly build a workflow from UI in Orkes Conductor.

For this,

  1. Navigate to Definitions > Workflow, and click the Define Workflow button.
  2. Create a workflow with an event task using the following JSON template:
{
"name": "event_task",
"taskReferenceName": "event_task_ref",
"inputParameters": {
"param": "value"
},
"type": "EVENT",
"decisionCases": {},
"defaultCase": [],
"forkTasks": [],
"startDelay": 0,
"joinOn": [],
"sink": "sqs:arn:aws:sqs:us-west-2:606699376960:crossaccountqueue",
"optional": false,
"defaultExclusiveJoinTask": [],
"asyncComplete": false,
"loopOver": [],
"onStateChange": {}
}
  1. The event task should use a sink of “sqs:full-queue-arn” or a sink of “sqs:https://full-queue-url”.
note

Any usage of SQS in the “event” field of the event handler or the “sink” field of the event task must use queue URL or queue ARN.

You can also use the following API to create the workflow:

POST /api/metadata/workflow

Check out the API doc for more details about creating a workflow using the API method.

Executing Workflow in Orkes Conductor

The workflow can be run using different methods. For quick testing, you can use the Run Workflow button on the left menu on your Orkes Conductor console.

Running workflow from Conductor UI

Or you can use the following API to start the workflow execution:

POST /api/workflow/{name}

Provide the workflow name as the input parameter and run the command.

The response body of the API request returns the workflow execution ID.

Workflow execution ID returned in the response body

You can verify the execution status by navigating to Executions > Workflows from the Conductor UI and searching using this workflow ID.

Workflow execution from Conductor UI

Upon successful execution, go to AWS Web Console for that SQS queue, poll for messages, and inspect the payload to verify the integration is successful.

Verifying from AWS console