Tasks in Workflows

When building workflows, you can use the built-in system tasks and operators provided by Conductor and additionally write your own custom worker tasks to implement custom logic.

Built-in tasks

Here is an introduction of the built-in tasks available in Conductor. These system tasks and operators enable you to quickly build workflows without needing to code.

• System tasks are managed by Conductor and run within its JVM (Java Virtual Machine), allowing you to get started quickly without needing custom workers. System tasks include AI tasks that can be used to build AI-powered and agentic applications.
• Operators are also managed by Conductor and run within its JVM (Java Virtual Machine). They enable you to declaratively design the workflow control flow and logic with minimal code required.

For control flow

The control structures and operations in your Conductor workflow are implemented as tasks. Here are the tasks available for managing the flow of execution:

• Conditional flow

  • Switch—Execute tasks conditionally, like an if…else… statement.

• Looping flow

  • Do While—Execute tasks repeatedly, like a do…while… statement.

• Parallel flows

  • Fork—Execute a static number of tasks in parallel.
  • Dynamic Fork—Execute a dynamic number of tasks in parallel.
  • Join—Join the forks after a Fork or Dynamic Fork before proceeding to the next task.
  • Start Workflow—Asynchronously start another workflow.

• Jumps or state changes in flow

  • Terminate—Terminate the current workflow, like a return statement.
  • Sub Workflow—Synchronously start another workflow, like a subroutine.
  • Terminate Workflow—Terminate another ongoing workflow.
  • Update Task—Update the status of another ongoing task.

• State querying

  • Get Workflow—Get the execution details of another ongoing workflow.

• Waits in flow

  • Wait—Pause the current workflow until a set time, duration, or signal is received.
  • Wait for Webhook—Pause the current workflow for an incoming webhook signal.
  • Human–Pause the current workflow for human input before proceeding to the next task.
  • Yield–Temporarily yields control over the workflow by pausing the execution until it is explicitly resumed using specific endpoints.

• Dynamic tasks in flow

  • Dynamic—Execute a task dynamically, like a function pointer.

For assigning variables

In general, variables are bounded within each task and passed along in the workflow as necessary. However, you can also handle variables or secrets at a global environment or workflow level.

• Set Variable—Create or update workflow variables.
• Update Secret—Create or update secrets in your Conductor cluster.

For execution logic

In most common cases, you can make use of existing Conductor features instead of creating a custom worker from scratch. These include tasks for data transformation, user journeys, and LLM chaining.

Use Case	Task to Use
Call an API or HTTP endpoint	HTTP
Poll an API or HTTP endpoint	HTTP Poll
Publish or consume events	Event
Clean or transform JSON data	JSON JQ
Modify SQL databases	JDBC
Execute JavaScript scripts	Inline
Send email using SendGrid integration	SendGrid
Evaluate and retrieve data in spreadsheets	Business Rule
Get authorized using a signed JWT	Get Signed JWT
Orchestrate human input in the loop	Human
Query data from Conductor Search API or Metrics	Query Processor
Send alerts to Opsgenie	Opsgenie
Retrieve text or JSON content from a URL	Get Document
Generate text embeddings	Generate Embeddings
Store text embeddings in a vector database	Store Embeddings
Generate and store text embeddings in a vector database	Index Text
Chunk, generate, and store text embeddings in a vector database	Index Document
Retrieve data from a vector database	Get Embeddings
Retrieve data from a vector database based on a search query	Search Index
Generate text from an LLM based on a defined prompt	Text Complete
Generate text from an LLM based on a user query and additional system/assistant instructions	Chat Complete

Custom tasks

A Worker task can be used to implement custom logic outside the scope of Conductor’s system tasks. These tasks can be written in any programming language of your choice (Python, Java, JavaScript, C#, Go, and Clojure). Unlike a built-in task, a Worker task requires setting up a worker outside the Conductor environment that polls for and executes the task.

Refer to Writing Workers for Conductor Workflows for more information.

Task definition

A task definition specifies a task’s general implementation details, like the expected input and output keys, and failure-handling configurations, like rate limits, retries, and timeouts. This definition applies to all instances of the task across workflows.

All task definitions are stored as JSON. These parameters can be updated in real time without needing to redeploy your application.

Example

Here is an example task definition JSON:

{
  "createTime": 1721901586970,
  "updateTime": 1725926875230,
  "createdBy": "user@acme.com",
  "updatedBy": "user@acme.com",
  "name": "calculate-fx",
  "description": "Calculates currency exchange",
  "retryCount": 0,
  "timeoutSeconds": 3600,
  "inputKeys": [],
  "outputKeys": [],
  "timeoutPolicy": "TIME_OUT_WF",
  "retryLogic": "EXPONENTIAL_BACKOFF",
  "retryDelaySeconds": 30,
  "responseTimeoutSeconds": 600,
  "concurrentExecLimit": 20,
  "inputTemplate": {},
  "rateLimitPerFrequency": 10,
  "rateLimitFrequencyInSeconds": 1,
  "ownerEmail": "user@acme.com",
  "pollTimeoutSeconds": 3600,
  "backoffScaleFactor": 1,
  "enforceSchema": false
}

Before adding a Worker task or a Human task in a workflow, it must be registered as a task definition in Conductor.

Task definitions enable configuration of the task’s failure handling settings and RBAC governance. You may also choose to create a task definition for other system tasks if required to configure task-specific retry settings, timeout settings, rate-limit settings, and more.

Task definitions can be registered via the Conductor UI or through SDK/API. Once registered, they can be referenced and used in different workflows.

Task configuration

The task configuration is a task’s configuration details, which are part of the workflow definition. It specifies the workflow-specific implementation details, like the task reference name, task type, task input parameters, and so on.

Although each task type has its unique configuration, all tasks share several parameters in common.

• For all tasks, the configuration specifies the input parameters for the task.
• For custom worker tasks, the configuration contains a reference to a registered worker task definition.
• For system tasks and operators, the configuration includes parameters that control the task behavior. For example, the configuration for an HTTP task specifies the endpoint URL and the payload template, which will be used during task execution.

Refer to the Task Reference to learn more about the task configuration for each task type.

Common configuration parameters

The task configurations appear in the tasks array of the workflow definition JSON. For example:

{
 "name": "WorkflowDefinition",
 "description": "Workflow definition",
 "version": 1,
 "tasks": [], // The task configuration appears here 
 "inputParameters": [],
 "outputParameters": {},
 "schemaVersion": 2,
 "restartable": true,
 "workflowStatusListenerEnabled": false,
 "ownerEmail": "john.doe@acme.com",
 "timeoutPolicy": "ALERT_ONLY",
 "timeoutSeconds": 0,
 "failureWorkflow": ""
}

Each task configuration JSON object may contain the following parameters:

Parameter	Description	Required/ Optional
name	Name of the task. The default value is the same as the task type. The name can be changed to something descriptive. To use a given task definition, the task name here must match the task definition name (case-sensitive). Note: It is recommended to use alphanumeric characters for task names. While special characters are allowed for backward compatibility, they are not fully supported and may cause unexpected behavior.	Required.
taskReferenceName	Reference name for the task. Must be a unique value in a given workflow.	Required.
type	The task type. For example, HTTP, SIMPLE.	Required.
inputParameters	Map of the task’s input parameters.	Depends on the task type.
optional	Whether the task is optional. If set totrue, any task failure is ignored, and the workflow continues with the task status updated to COMPLETED_WITH_ERRORS. However, the task must reach a terminal state. If the task remains incomplete, the workflow waits until it reaches a terminal state before proceeding.	Optional.
asyncComplete	Whether the task is completed asynchronously. The default value is false. Supported values: false—Task status is set to COMPLETED upon successful execution. true—Task status is kept as IN_PROGRESS until an external event completes it.	Optional.
startDelay	The time in seconds to wait before making the task available for worker polling. The default value is 0.	Optional.
onStateChange	Configuration for publishing an event when the task status changes.	Optional.

Dealing with data

Conductor provide several mechanisms to pass, validate, and protect data as it moves between tasks. This section covers how to pass data using dynamic references and input templates, mask sensitive values, and enforce schema validation for inputs and outputs.

Passing data between tasks

Using dynamic references, data can be passed from one task to another. These dynamic references are formatted as JSONPath expressions. Refer to Wiring Parameters to learn more.

Passing data using task input templates

Use the task input templates in a task definition to apply default parameters to all instances of the task. Refer to Using Task Input Templates to learn more.

Masking data in tasks

Masking parameters protects sensitive data from being exposed in workflows. It ensures that sensitive values are hidden and not displayed in the workflow definitions or executions. Refer to Masking Parameters to learn more.

Input/output schema validation

Create schemas to define and enforce the payload structure of workflow or task inputs/outputs. Refer to Input/Output Schema Validations to learn more.

Task reuse

Since task workers typically perform a unit of work as part of a larger workflow, Conductor’s infrastructure is built to enable task reusability out of the box. Once a task is defined in Conductor, it can be reused numerous times:

• In the same workflow, using different task reference names.
• Across various workflows.

When reusing tasks, it's important to think of situations that a multi-tenant system faces. By default, all the work assigned to this worker goes into the same task queue. This could result in your worker not being polled quickly if there is a noisy neighbor in the ecosystem. You can tackle this situation by scaling up the number of workers to handle the task load, or even using task-to-domain to route the task load into separate queues.