AWS S3 Integration with Orkes Conductor

Available since

• v5.3.0 and later

Integrating AWS S3 with Orkes Conductor lets you build workflows that interact with your AWS S3 using the following operations:

• Create Bucket
• List Buckets
• Search Buckets
• Delete Bucket
• Upload Object
• List Objects
• Download Object
• Get Object Metadata
• Copy Object
• Delete Object
• Create Folder
• List Folders
• Delete Folder

This guide explains how to set up and use the integration. Here’s an overview:

1. Get the required credentials from AWS.
2. Configure a new AWS S3 integration in Orkes Conductor.
3. Use AWS S3 integration in workflows.

Step 1: Get the AWS credentials

To integrate AWS S3 with Orkes Conductor, create an IAM user in AWS and generate access keys with the required S3 permissions.

To create an IAM user and generate access keys:

1. Log in to the AWS Console and search for IAM in the top search bar.
2. In the left navigation, go to Access Management > IAM Users.
3. Select an existing user or select Create user to create a new one.
4. Select the user, go to the Security credentials tab, scroll to Access keys, and select Create access key.
5. For Use case, select Application running outside AWS and complete the creation flow.
6. On the final screen, copy the Access key and Secret access key. Store these securely as the secret access key is shown only once.

To attach IAM policy to the user:

1. Go to Access Management > IAM Users and select your user.
2. Go to the Permissions tab, and select Add permissions > Create inline policy.
3. Switch to the JSON tab, and paste a policy granting at minimum: s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject, s3:CreateBucket, s3:DeleteBucket, s3:ListAllMyBuckets. For example:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"s3:ListBucket",
				"s3:GetObject",
				"s3:PutObject",
				"s3:DeleteObject",
				"s3:CreateBucket",
				"s3:DeleteBucket",
				"s3:ListAllMyBuckets"
			],
			"Resource": [
				"arn:aws:s3:::*",
				"arn:aws:s3:::*/*"
			]
		}
	]
}

4. Give the policy a name and select Create policy.

Step 2: Add an integration for AWS S3

After obtaining the credentials, add an AWS S3 integration to your Conductor cluster.

To create an integration:

1. Go to Integrations > Connected Apps from the left navigation menu on your Conductor cluster.
2. Select + New integration.
3. In the Cloud section, choose AWS S3.
4. Select + Add and enter the following parameters:

Parameter	Description	Required/Optional
Integration name	A name for the integration.	Required.
Access Key ID	The AWS Access Key ID generated from Step 1. If left empty, it uses any AWS credentials already configured in the environment (environment variables, credentials file, or IAM role).	Required.
Secret Access Key	The Secret Access Key generated from Step 1. If left empty, it uses any AWS credentials already configured in the environment (environment variables, credentials file, or IAM role).	Required.
Region	The AWS region of your S3 bucket. For example us-east-1.	Required.
Description	A description of the integration.	Required.

5. (Optional) Toggle the Active button off if you don’t want to activate the integration instantly.
6. Select Save.

The integration is created successfully.

Step 3: Use AWS S3 integration in workflows

Once the integration is ready, this can be used directly within the workflows.

To use AWS S3 integration in a workflow:

1. Go to Definitions > Workflow from the left navigation menu on your Conductor cluster.
2. Select + Define workflow.
3. In your workflow, select the (+) icon and select Connected Apps.

4. In Add Task panel on the right, search for AWS S3, and select the integration created in Step 2.

The following operations are available for use with this integration.

Operation	Description
Create Bucket	Creates a new S3 bucket.
List Buckets	Lists all S3 buckets in the AWS account.
Search Buckets	Searches for S3 buckets by partial name match.
Delete Bucket	Deletes an empty S3 bucket.
Upload Object	Uploads content as an S3 object to a specified bucket.
List Objects	Lists objects in an S3 bucket, optionally filtered by prefix.
Download Object	Downloads an S3 object and returns its content as a string.
Get Object Metadata	Retrieves metadata about an S3 object, including size, last modified date, and content type.
Copy Object	Copies an S3 object to a destination bucket in the same AWS region.
Delete Object	Deletes an object from an S3 bucket.
Create Folder	Creates an empty folder in an S3 bucket.
List Folders	Lists folders in an S3 bucket.
Delete Folder	Deletes a folder and all objects within it.

5. Select the required operation, configure the parameters, and select Save > Confirm.
6. Select Execute to run the workflow.

For the complete operations parameters and output reference, see AWS S3 Operation Reference.