Skip to main content

Workflows

What are Workflows?

At a high level, a workflow is the Conductor primitive that encompasses the definition and flow of your business logic. A workflow is a collection (graph) of tasks and sub-workflows. A workflow definition specifies the order of execution of these Tasks. It also specifies how data/state is passed from one task to the other (using the input/output parameters). These are then combined together to give you the final result. This orchestration of Tasks can happen in a hybrid ecosystem that includes microservices, serverless functions, and monolithic applications. They can also span across any public cloud and on-premise data center footprints. In addition, the orchestration of tasks can be across any programming language since Conductor is also language agnostic.

One key benefit of this approach is that you can build a complex application using simple and granular tasks that do not need to be aware of or keep track of the state of your application's execution flow. Conductor keeps track of the state, calls tasks in the right order (sequentially or in parallel, as defined by you), retry calls if needed, handle failure scenarios gracefully, and outputs the final result.

Leveraging workflows in Conductor enables developers to truly focus on their core mission - building their application code in the languages of their choice. Conductor does the heavy lifting associated with ensuring high reliability, transactional consistency, and long durability of their workflows. Simply put, wherever your application's component lives and whichever languages they were written in, you can build a workflow in Conductor to orchestrate their execution in a reliable & scalable manner.

What does a Workflow look like?

Let's start with a basic workflow and understand what are the different aspects of it. In particular, we will talk about two stages of a workflow, defining a workflow and executing a workflow

Simple Workflow Example

Assume your business logic is to simply to get some shipping information and then do the shipping. You start by logically partitioning them into two tasks

  • shipping_info
  • shipping_task

The next step is to create a workflow in Conductor using JSON and then create the associated tasks in Conductor for the above identified ones. f After that, you add those tasks into the workflow to have shipping_info called first and then, if it is successful, call shipping_task. You now have a definition of the workflow in Conductor and Conductor will then generate an easy to understand visual representation of this workflow

Simple Shipping Workflow - Visual Representation

Multiple Paths Workflow Example

Next let's see a more complex example where you want to support multiple shipping vendors (e.g. FedEx, DHL, UPS) and the code for each of them live in separate services. This is a good design pattern to follow since you can now independently change each of them without having to worry about breaking others. Usually this means you now have to take on the work of wiring up many different services into your primary execution path. But with Conductor, this just means adding a switch operator to decide which vendor to call depending on an incoming parameter, and then during execution time, the right one will be called!

Multi-vendor Shipping Workflow - Visual Representation of Design

Furthermore, with Conductor, in addition to the above design view of the workflow, you can also see the execution view of the workflow. In this particular example, the workflow picked UPS at runtime and as seen from the green color of the tasks in the execution path, this workflow was completed successfully. If a particular task had failed, it would be shown in red.

Multi-vendor Shipping Workflow - Visual Representation of Execution

The Power of Seeing

These visual representations of workflows are key to how Conductor turbocharges the productivity of engineering teams.

  • Workflows definitions serve as the enduring documentation for all the different applications a team owns and this benefit becomes even more powerful as the team scales in size and scope. Furthermore, it allows anyone new to the team to quickly get ramped up.
  • The execution visualization allows you to quickly identify problem areas and provides you details on the error responses received, details on where the failing task was executing etc. This makes debugging much faster than digging across distributed logs and events making Conductor's approach to workflows relevant not only during the creation time but also during live operations of the workflows in production.

How do I use Workflows?

Starting Workflows

Once a workflow is defined in Conductor, it is ready to be invoked. An invocation executes the workflow and passes in any arguments that were provided by the caller. There are three ways in which a workflow can be invoked.

View Workflows

Once a workflow is invoked, it starts running and you can view details of its execution status

Update Workflows

When your application's business logic evolves or you need to fix an error in your workflow definition, you can update your workflows in Conductor with built-in support for versioning.

The Power of Versioning

Conductor's native support for versioning allows developers to rapidly iterate on new features even with multiple invocations of the same workflow are in-flight. Unlike other platforms where you either need to wait till those in-flight executions finish or forcefully error them out, with Conductor you can have both versions in-flight at the same time. In addition to increasing developer agility, this also unlocks other benefits

  • Experimenting new features for a small subset of users
  • Safely test changes in production while containing the blast radius to a known value

Further Reading