Skip to main content

Step 6: Executing Tasks in Parallel

Continuing from Step 5

Another exciting feature of Conductor is executing tasks in parallel. A common scenario is to run a specific task based on an array of values. Let's try this out on our workflow.

To do this, we will use the payment deposit workflow we created so far as a sub-workflow. In a new workflow, we will run a task that will return a given number of deposit transactions. For each transaction, we will trigger this sub-workflow - which will all run in parallel.

tip

Orkes Conductor runs 1000s of parallel executions, limited only by the worker capacity. The more workers you have, the more parallelism you can have.

  1. In your current workflow, add a new workflow called batch-process-payments-<unique-id> and add a Worker task for the case with the name retrieve-deposit-batch.
  2. Add a Dynamic Fork task - and configure it to run a sub-workflow based on the output of retrieve-deposit-batch task.
  3. Run workflow directly from the UI using the Run Workflow button.

If we named the task retrieve-deposit-batch, we'd notice that it is actually executed (in playground env), but how? That's because there is a pre-defined task that is polling and running all the tasks named retrieve-deposit-batch. We also have the required permissions in the playground for this task.

Here is the code reference for this worker:

Complete source file on Github: .../workers/ConductorWorkers.java

    @WorkerTask(value = "retrieve-deposit-batch", threadCount = 5, pollingInterval = 200)
    public List<DepositDetail> retrieveDepositBatch(@InputParam("batchCount") Integer batchCount) {
        if (batchCount == null) {
            batchCount = random.nextInt(5, 11);
        }
        batchCount = Math.min(100, batchCount); // Limit to 100 in playground
        List<DepositDetail> depositDetails = IntStream.range(0, batchCount)
                .mapToObj(i -> DepositDetail.builder()
                        .accountId("acc-id-" + i)
                        .amount(BigDecimal.valueOf(i * 1500L)) // Create random amounts
                        .build())
                .toList();
        log.info("Returning {} transactions", depositDetails.size());
        return depositDetails;
    }

By default, it would return a random value between 5-10 tasks. If we supply an input called batchCount - we can retrieve up to 100 transactions. This is a limitation of the playground. In a dedicated cluster, you can run parallel tasks into the thousands or tens of thousands depending on the cluster's capacity.

Try out a larger batch

Try configuring a larger batch using the input called batchCount to the task retrieve-deposit-batch task - we can observe the tasks running in parallel by looking at the timeline view or task list view.