CI/CD Best Practices

Conductor resources and metadata are the core part of your application and should be versioned, controlled, and released similarly to the code.

When moving Conductor resources across different environments, there are three main components involved:

• Workflow definitions (maintained as JSON or as code)
• Other metadata, like task definitions, user forms, event handlers, webhooks, AI prompts, schemas, and so on (maintained as JSON)
• Worker implementation used by the workflows (in code)

CI/CD for Conductor workflows and resources

Similar to any CI/CD pipeline, this is the general process for managing Conductor resources across releases:

1. Download the resource JSONs to a version control system.
2. (For workflow definitions) Run tests on the changes.
3. Deploy the resource JSONs to the target environment using a pipeline or script.

The exact steps depend on your own CI/CD implementation, but this guide will showcase some general procedures to accomplish this.

Step 1: Download resource JSONs

Use the Conductor UI to download each JSON one by one, or the API to download the JSONs in bulk.

Conductor UI

To download each JSON:

1. Go to Definitions > Workflows or Task or User Forms or Event Handler or Webhook or AI Prompts or Schemas in the left navigation menu.
2. Select the desired resource.
3. Retrieve the JSON in one of the following ways:
   • Select Download to download the .json file, or
   • Go to the Code tab and copy the JSON into a file.
4. Save the files to your version control system.

API

Prerequisites

You must generate an access token to authenticate your API requests. Refer to Authentication for more information.

• To retrieve the workflow definition JSONs, you can use the Get All Workflow Definitions API, which allows you to filter based on tags or a specific workflow.
• To retrieve the task definition JSONs, you can use the Get All Task Definitions API, which also allows you to filter based on tags.
• To retrieve the user form JSONs, you can use the Get User Forms API.

Refer to the API Docs on your Conductor UI for more details on the available API endpoints.

Once the resources are updated in your version control system, you can open a PR and begin testing.

Step 2: Test workflow definitions

As part of the CI pipeline, workflow definitions should be maintained as independent units that can be unit and integration-tested before being released to the production environment.

For a guide on testing Conductor workflows, refer to Unit and Regression Tests.

Step 3: Deploy changes to the target environment

When deploying the changes to your target environment, it is crucial to assess what resources are being modified and the deployment scope. For example, the deployment will only target resources grouped under a specific tag, a file name prefix, etc.

Once you have determined the deployment scope, map the resources to the endpoints you need to call to deploy the changes. For example, use the Update Workflow Definition API to update the workflow definition JSONs.

tip

Refer to the API Docs on your Conductor UI for the full list of available API endpoints for updating the different resources.

You can use pipelines or deployment scripts to push the changes in an automated fashion.

Example deployment script

The following script cycles through all the workflow definitions in the current folder and uploads them to the Conductor server.

To use the script, you must inject ORKES_ACCESS_KEY, ORKES_ACCESS_SECRET, and ORKES_CONDUCTOR_SERVER_URL (ending in /api) variables with the appropriate values in your deployment pipeline. Refer to Authentication for a guide on retrieving your Orkes access tokens.

export response=`curl -s -X POST $CONDUCTOR_SERVER_URL/token -H 'Content-Type:application/json' -d '{
	"keyId": "'"$CONDUCTOR_AUTH_KEY"'",
	"keySecret": "'"$CONDUCTOR_AUTH_SECRET"'"
}'`

if [[ "$response" != *'token'* ]]; then
 echo "Unable to generate the auth header.  Please check KEY, SECRET and CONDUCTOR_SERVER_URL variables"
 echo "Server response:"
 echo $response
 exit 1
fi

export token=`echo $response | cut -d '"' -f4`

for FILE in main/resources/workflows/*;
 do
   echo "Deploying @$FILE";

   curl -X POST $CONDUCTOR_SERVER_URL/metadata/workflow?overwrite=true          -H "X-Authorization: $token"          -H "accept: */*"          -H "Content-Type: application/json"          -d @$FILE
 done

Refer to the full source code in .../src/deploy_workflows.sh.

CI/CD for workers

Workers are application-specific code deployed in a non-Conductor environment. They should be maintained, tested, and released as any other code released to production. Here are some best practices for maintaining workers:

• Keep worker deployments and maintenance separate from the workflows.
• Unit test workers based on the expected inputs and outputs, similar to any other application code.
• Workers are the unit of scale for your workflows. You can either deploy each worker in an independent container or group a set of workers to be scaled up/down together.