Skip to main content

Retries, Failures and Rate Limits

Task Rate Limits

There are times where a task should not be run more than a certain rate. A rate limit for the task can be set with the following two parameters:

  • rateLimitFrequencyInSeconds: sets the number of seconds for the rate limit.
  • rateLimitPerFrequency: sets the volume of requests that can be completed over the timeframe.

For example, this will only allow the task to be run 100 times in a minute:

"rateLimitFrequencyInSeconds": 60,
"rateLimitPerFrequency": 100

Task Timeout

Tasks can timeout after a set period. This timeframe can be set with the timeoutSeconds parameter in the task. If the task should never time out, this can be set to 0.

When a task times out, the timeoutPolicy defines what should happen next. The possible parameters are:

  • RETRY: Run the task again.
  • TIME_OUT_WF: workflow is marked ast TIMED_OUT and terminated.
  • ALERT_ONLY: Registers a counter (task_timeout)

Task Retries

If a task should fail, it can be set to retry using the retryLogic parameter. The two options are FIXED and EXPONENTIAL_BACKOFF.

The timing of the retry defaults to 60 seconds, but can be explicitly set with retryDelaySeconds, adn the number of retries is default 3, but can be set with retryCount.

If the task does not successfully complete after the set number of retries, the task will fail.

Task Failure

When a task has failed to execute (due to timeout or other failure), the task will change status to FAILED. When a task fails, the entire workflow is set as FAILED.

Workflow failure

When a workflow fails, there are two ways to handle the exception:

  • Create a failureWorkflow that runs on workflow failure (see detailed documentation)
  • Create a custom implementation of the Workflow Status Listener, and add setWorkflowStatusListenerEnabled to your workflow (more details).

Rate limiting your workflow

In order to not harm your workers or other downstream projects, it may be required to rate limit your workflows. This can be done by

TODO