Skip to main content

CI/CD Best Practices

Conductor workflows are the core part of your application and should be versioned, controlled, and released similarly to the code.

Conductor Workflows​

Workflows have two main components:

  1. Workflow definition that is maintained as a file (JSON or code)
  2. Worker implementation used by the workflows

Workflows​

Workflow definitions should be maintained as an independent unit that can be unit and integration tested before releasing to the production environment.

For details on how to unit test workflows, see Unit and Regression Testing Workflows.

Steps to publish your workflows as part of CI/CD​

Notes

See Generating Tokens (Video) on how to generate an access token for the API requests below.

Downloading workflows from Conductor cluster to check into your version control​

  1. Download the JSON from the Conductor UI (Use the download button on the definition page).
  2. (Alternatively) Use the API to download JSON for the workflow. 
    # Get the workflow definition given name and version
    GET -H "X-Authorization:<TOKEN>" /api/metadata/workflow/{name}?version=<version>
    # Retrieve all the workflows
    GET -H "X-Authorization:<TOKEN>" /api/metadata/workflow/

Publish workflows to Conductor cluster from your deployment scripts​

Inject ORKES_ACCESS_KEY, ORKES_ACCESS_SECRET and ORKES_CONDUCTOR_SERVER_URL variables with the appropriate values in your deployment pipeline.

The following script cycles through all the workflow definitions in the current folder and uploads it to the Conductor server.

Complete source file on Github: .../src/deploy_workflows.sh
export response=`curl -s -X POST $CONDUCTOR_SERVER_URL/token -H 'Content-Type:application/json' -d '{
        "keyId": "'"$CONDUCTOR_AUTH_KEY"'",
        "keySecret": "'"$CONDUCTOR_AUTH_SECRET"'"
}'`

if [["$response" != *'token'*]]; then
echo "Unable to generate the auth header. Please check KEY, SECRET and CONDUCTOR_SERVER_URL variables"
echo "Server response:"
echo $response
exit 1
fi

export token=`echo $response | cut -d '"' -f4`

for FILE in main/resources/workflows/*;
do
echo "Deploying @$FILE"

curl -X POST $CONDUCTOR_SERVER_URL/metadata/workflow?overwrite=true   -H "X-Authorization: $token"   -H "accept: */*"   -H "Content-Type: application/json"   -d @$FILE
done

CI/CD for workers​

Workers are application-specific code and should be maintained, tested, and released as any other code released to production.

Best practices for maintaining workers​

  1. Keep worker deployments and maintenance separate from the workflows.
  2. Unit test workers based on the expected inputs and outputs similar to any other application code.
  3. Workers are the unit of scale for your workflows. Either deploy each worker in an independent container or group set of workers that typically need to be scaled up/down together.