Retries, Failures and Rate Limits
Task Rate Limits
There are times where a task should not be run more than a certain rate. A rate limit for the task can be set with the following two parameters:
rateLimitFrequencyInSeconds: sets the number of seconds for the rate limit.
rateLimitPerFrequency: sets the volume of requests that can be completed over the timeframe.
For example, this will only allow the task to be run 100 times in a minute:
Tasks can timeout after a set period. This timeframe can be set with the
timeoutSeconds parameter in the task. If the task should never time out, this can be set to 0.
When a task times out, the
timeoutPolicy defines what should happen next. The possible parameters are:
RETRY: Run the task again.
TIME_OUT_WF: workflow is marked ast TIMED_OUT and terminated.
ALERT_ONLY: Registers a counter (task_timeout)
If a task should fail, it can be set to retry using the
retryLogic parameter. The two options are
The timing of the retry defaults to 60 seconds, but can be explicitly set with
retryDelaySeconds, adn the number of retries is default 3, but can be set with
If the task does not successfully complete after the set number of retries, the task will fail.
When a task has failed to execute (due to timeout or other failure), the task will change status to FAILED. When a task fails, the entire workflow is set as FAILED.
When a workflow fails, there are two ways to handle the exception:
- Create a
failureWorkflowthat runs on workflow failure (see detailed documentation)
- Create a custom implementation of the Workflow Status Listener, and add
setWorkflowStatusListenerEnabledto your workflow (more details).
Rate limiting your workflow
In order to not harm your workers or other downstream projects, it may be required to rate limit your workflows. This can be done by