Retries, Failures and Rate Limits
Task Rate Limits
Sometimes, a task should not be run at more than a certain rate. A rate limit for the task can be set with the following two parameters:
rateLimitFrequencyInSeconds: Specifies the number of seconds for the rate limit.
rateLimitPerFrequency: Specifies the volume of requests that can be completed over the timeframe.
For example, this will only allow the task to be run 100 times in a minute:
Tasks can timeout after a set period. This timeframe can be set with the
timeoutSeconds parameter in the task. If the task should never time out, this can be set to 0.
When a task times out, the
timeoutPolicy defines what should happen next. The possible parameters are:
RETRY: Run the task again.
TIME_OUT_WF: The workflow is marked as TIMED_OUT and terminated.
ALERT_ONLY: Registers a counter (task_timeout)
If a task should fail, it can be set to retry using the
retryLogic parameter. The two options are
The retry timing defaults to 60 seconds but can be explicitly set with
retryDelaySeconds, and the number of retries is default 3, but can be set with
If the task does not successfully complete after the set number of retries, the task will fail.
When a task has failed to execute (due to timeout or other failures), the task will change status to FAILED. When a task fails, the entire workflow is set as FAILED.
When a workflow fails, there are two ways to handle the exception:
- Create a
failureWorkflowthat runs on workflow failure (see detailed documentation)
- Create a custom implementation of the Workflow Status Listener, and add
setWorkflowStatusListenerEnabledto your workflow (more details).
Rate limiting your workflow
In order to not harm your workers or other downstream projects, it may be required to rate limit your workflows.