Skip to main content

Retries, Failures and Rate Limits

Task Rate Limits​

Sometimes, a task should not be run at more than a certain rate. A rate limit for the task can be set with the following two parameters:

  • rateLimitFrequencyInSeconds: Specifies the number of seconds for the rate limit.
  • rateLimitPerFrequency: Specifies the volume of requests that can be completed over the timeframe.

For example, this will only allow the task to be run 100 times in a minute:

"rateLimitFrequencyInSeconds": 60,
"rateLimitPerFrequency": 100

Task Timeout​

Tasks can timeout after a set period. This timeframe can be set with the timeoutSeconds parameter in the task. If the task should never time out, this can be set to 0.

When a task times out, the timeoutPolicy defines what should happen next. The possible parameters are:

  • RETRY: Run the task again.
  • TIME_OUT_WF: The workflow is marked as TIMED_OUT and terminated.
  • ALERT_ONLY: Registers a counter (task_timeout)

Task Retries​

If a task should fail, it can be set to retry using the retryLogic parameter. The two options are FIXED and EXPONENTIAL_BACKOFF.

The retry timing defaults to 60 seconds but can be explicitly set with retryDelaySeconds, and the number of retries is default 3, but can be set with retryCount.

If the task does not successfully complete after the set number of retries, the task will fail.

Task Failure​

When a task has failed to execute (due to timeout or other failures), the task will change status to FAILED. When a task fails, the entire workflow is set as FAILED.

Workflow failure​

When a workflow fails, there are two ways to handle the exception:

  • Create a failureWorkflow that runs on workflow failure (see detailed documentation)
  • Create a custom implementation of the Workflow Status Listener, and add setWorkflowStatusListenerEnabled to your workflow (more details).

Rate limiting your workflow​

In order to not harm your workers or other downstream projects, it may be required to rate limit your workflows.