Rate Limits for Workflows

Announcements Product News
,
1

Announcing Workflows Rate Limits

To enhance your experience with SailPoint’s Identity Security Cloud, we are introducing updates to SaaS workflows starting January 16, 2025. Some workflows generate an exceptionally high number of executions daily, causing system-wide lag and impacting service quality for all tenants. To maintain system stability and equitable performance, we’re implementing standardized workflow execution rate limits that are fair and sufficient for meeting customer business needs.

New Workflow Execution Terms:

  • Initial Allowance: Each tenant can execute up to 400,000 workflows per day at the current rate.

  • Rate Limiting Beyond the Limit: After reaching the 400,000-execution threshold, workflow executions will be limited to 5 invocations per second.

  • Quota Reset: The execution quota resets every 24 hours.

Optimizing Your Workflows:

High execution rates can slow down services for tenants in the same region, affecting processing times across the entire customer community. Optimizing your workflows can significantly reduce unnecessary executions and can be done in a few easy steps. To optimize your workflows:

  1. Use Filters: Apply filters to trigger (events that start workflows) a workflow only when specific conditions are met.

  2. Review Best Practices: Visit this link for tips on streamlining workflows to align with your business needs.

Why This Matters:

By implementing workflow rate limiting, we aim to have fair, consistent, and reliable service for all customers. These limits provide an opportunity to evaluate and refine your workflows, ensuring they operate efficiently with the right balance of steps and executions aligned to your business goals.

If you have any questions, please reach out to your Customer Success Manager or Support Expert for assistance. You may also leverage the “Provide Feedback” link in Identity Security Cloud or SailPoint’s Ideas Portal.

4 Likes
2

Hi Colin!

Is there an alert, dashboard, or API endpoint that will let us know the number of executions used in the current period, or tell us when we’re approaching/have hit the limit?

1 Like
3

I was also going to ask the same as Mark. Hoping that there’s some way to see total number of executions in a day etc

4

Although it sounds like a reasonable thing to try and limit the amount of unnecessary computations, which could benefit us all, I also have my doubts:

Announcement too late
You announced this on the 16th of January 2025 and say this functionality goes into production on the 16th of January 2025. If organizations are already surpassing this limit, they are in for a unpleasant surprise aren’t they?

Fairness
I doubt that the current strategy is considered fair.
If organization A has 100 times as many identities and 10 times as many applications compared to organization B. Wouldn’t it be considered fair that organization A gets a higher limit? After all I can imagine more workflows need to be executed there, and I am not involved with license costs, but I doubt these will be equal as well.

workflow execution metrics
In addition, some of our workflows we can’t see the number of executions as the POST /v2024/workflow-metrics API is returning a 502 bad gateway error for some of the workflows (and not necessarily the ones we expect high numbers for).
Also this API is undocumented (@christina_gagnon).

Besides that, I don’t believe there is an API to see the amount of daily executions, not per workflow, but also not in total. Making it more difficult to see if we pass that number, to ensure we stay under it.

Spikes
And finally, I can imagine that occasional activities like a migration, onboarding of a new source with 400k+ accounts can cause one-time spikes of executions. If we usually have 10k executions per day and require 1000k at the start of each month, we get a problem, even though this will be less executions than if one has 100k executions each day. It will require workarounds and can cause delays to spread these type of executions over multiple days.

Alternative improvements
Since workflows are mainly there to compensate for functionality that SailPoint does not offer out of the box or in a desired manner, putting a limit like this could limit what we can do.
In addition some workflows take many executions just because of limited SailPoint functionality. For example since you can not filter over the pending approvals with
a filter like created gt .... AND created lt .... As a consequence our workflow needs to iterate over them all. And since SailPoint APIs use pagination and workflows can not apply recursion (unless executing another workflow), this increases the number of executions as well.

Definition of amount of executions
Many of our workflows require looping over arrays. I know that in the backend, these are considered child workflows of dynamically created workflows. If we have a workflow where we loop over the owned roles of a single identity, and the identity has 40 owned roles, will this count as 41 executions (one for the parent workflow, 40 for the child workflow executions), or just as 1?

Passing the limit
What happens if we reach the limit? Will an error appear in the event logs for each failed workflow execution, perhaps such that we can retry them during a less busy day? How can we see which ones we missed and which workflows actually occurred?

Notification
We can get (email) notifications if sources get unhealthy.
Before this limitation you just announced gets added into production (although I think this is already to late due to the late announcement). Can we also get email notifications if we trigger this limit and email notifications in increasing urgency if we are approaching this limit? (Let’s say 50%, 75% and 90%?). If SailPoint is just suddenly cutting off workflows at a certain limit this could have critical effects, especially relating to production environments.

2 Likes
5

One way this is handled by some other platforms is through API credits. For periods where you’re well below your limit, you earn a certain number of credits which you can bank for a specified duration. If you then get really bursty all of the sudden, and blow past your limit, you can automatically spend some of your credits to avoid being throttled. Once all credits are expended, though, throttle away. This would be a good way to handle @angelo_mekenkamp’s scenario.

It would also be good if there was a way to request burst capacity via a support ticket in situations where, like in his other scenario, we’re onboarding a new source or maybe processing a merger or acquisition and have a ton of new identities being onboarded as a one-time load. Put in a support ticket and ask for extra workflow capacity a few days in advance, maybe work with support to schedule the load for a certain time window, etc.

We as customers don’t have the data to back up our concerns here, obviously, and I’m sure that SailPoint has identified the biggest offenders here and ensured that this isn’t going to cause major disruption for the majority of folks. That said, it could be helpful for us if SailPoint could provide some statistics on how many executions the different-sized environments with 5, 25, 50, 100 workflows average on a typical day, and how many customers in those cohorts are exceeding these new limits and could be negatively impacted by this change.

1 Like
6

I have asked the workflows PM to weigh in on this feedback, but I would like to reiterate that Workflow executions will not be dropped. Executions over the daily limit will be put in a queue for slower processing.

As for the other points of concern, our workflows PM will provide feedback.

3 Likes