Skip to main content

Quickstart 1: Write a Worker

Estimated time: 5min

As you will have learned in Core Concepts, the core units of every Conductor workflow are tasks and operators. All code is executed by workers; system tasks are serviced by internal Conductor workers, and any custom tasks must be serviced by workers of your own hosting.

Diagram of system workers (i.e. inline tasks, http tasks) and customer workers.

Custom workers can be deployed anywhere: container, VM, or bare metal.

In this quickstart, you will:

  1. Download a worker project.
  2. Integrate the worker with Conductor.
  3. Deploy the worker from your local machine.

Download a worker project​

Begin by creating a task worker that polls the Conductor server for scheduled tasks at regular intervals. To get started quickly, download one of our sample myTask worker projects in your preferred language:

git clone https://github.com/conductor-oss/awesome-conductor-apps.git

Integrate the worker with Conductor​

To connect your task worker with the Conductor server, you must:

  1. Register your worker by creating a task definition in Conductor
  2. Create a worker application and grant it Execute permission

Register worker​

To register your worker:

  1. Log in to your Conductor cluster or Developer Edition.
  2. In the left navigation menu, go to Definitions > Task.
  3. Select + Define task.
  4. Enter the Name for the task, which must match the task definition name in your worker code. This must be myTask if you are using the sample worker project downloaded in the previous step.
  5. Select Save > Confirm Save.

The task is now saved to the Conductor server, which facilitates the routing of the task to the correct worker pool during workflow execution.

Create a worker application​

Your worker requires programmatic access to the Conductor server. This is done by creating an application in Conductor; an access layer with its own permissions and access tokens.

To create an application for your worker:

  1. In the left navigation menu, go to Access Control > Applications.
  2. Select + Create application and enter a Name for it. For example, myApp or myWorkerApp.
  3. Enable the Worker application role, which allows the application to poll and update tasks.
  4. Select + Create access key and store the generated credentials securely.
  5. (Skip this step if you are using Developer Edition) Grant Execute permission to the application:
    1. Under Permissions, select Add permission.
    2. Select the Task tab and choose myTask.
    3. Enable the Execute toggle.
    4. Select Add Permissions.
  6. Now, go back to your worker project and set the credentials in the worker project file for your language:

Update the following in awesome-conductor-apps/python/developer-guides/using-workers/main.py. Note that base_url should not include /api.

    configuration = Configuration(
        base_url='https://your-cluster-url',
        authentication_settings=AuthenticationSettings(
            key_id='<YOUR-KEY-ID>',
            key_secret='<YOUR-KEY-SECRET>'
        )
    )

The application account can now execute the worker task.

Start the worker​

Using the command line, navigate to the downloaded project directory and launch the worker.

cd awesome-conductor-apps/python/developer-guides/using-workers
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
python3 main.py

When the worker starts successfully, you should see polling logs in the terminal. Later, when you run a workflow containing this worker, the task should run to completion. But first, let’s build a workflow using this task in Quickstart 2.