Troubleshooting 429 Error: Rate Limit Quota Violation in SailPoint IIQ Webservice Application

IdentityIQ (IIQ) IIQ Discussion and Questions
1

Hi All,

Hope all doing well.

I am working on a Webservice application in SailPoint IdentityIQ. I have one parent endpoint and three child endpoints derived from it.

During full account aggregation, I am experiencing an issue related to rate limiting. Despite my attempts to resolve this, including adding the following entries to the application XML:

<entry key="retryWait" value="30"/>
<entry key="retryableErrors">
  <value>
    <List>
      <String>429</String>
      <String>Too Many Requests</String>
      <String>Rate limit quota violation</String>
      <String>UsersAndDevices-V1_Default</String>
      <String>401</String>
      <String>Unauthorized</String>
    </List>
  </value>
</entry>

I also implemented a wait period in the “Before Operation” rule:

try {
  // Sleep for 200 milliseconds between each sequential child call
  Thread.sleep(200); 
} catch (InterruptedException e) {
  log.error("Throttling delay interrupted: " + e.getMessage());
}

However, I continue to receive the following error:

{
    "detail": "Rate limit quota violation. Quota limit exceeded. Identifier: UsersAndDevices-V1_Default.",
    "status": 429,
    "title": "Too Many Requests"
}

Exception during aggregation:
Reason: java.lang.RuntimeException: sailpoint.connector.ConnectorException:
Url: https://sbus.api.blackline.com/v1/users/210/entities,
Message: 429 : { “detail”: “Rate limit quota violation. Quota limit exceeded. Identifier: UsersAndDevices-V1_Default.”, “status”: 429, “title”: “Too Many Requests” }, HTTP Error Code: 429

I have attached the application XML and the “Get All Users” response for your reference.

Please help me to resolve this issue. Thank you in advance.

Regards,

Venu

All_Users_Response.xml (279.0 KB)

BlackLine_Tr_New.xml (61.8 KB)

2

@Venu9000 How much is the API limit?

3

Global Rate Limiting of 1000 requests/min: This equals roughly 16 requests per second globally across whole application client.

4

Looking at your xml , the root cause is simple: for every user, 3 child calls fire back-to-back with only 200ms between them. That’s too fast. What i would suggest is to increase the thread sleeping time in your before operation rule , can you try it and let me know, if it fixed your issue.

try {
    String endpointName = (String) requestEndPoint.get("uniqueNameForEndPoint");
    
    // Only 3 per-user child endpoints
    if ("Account Aggregation2".equals(endpointName) || 
        "Account Aggregation3".equals(endpointName) || 
        "Account Aggregation4".equals(endpointName)) {
        
        Thread.sleep(2000); // 2 seconds between each child call , since 200 is very little
        log.error("applied for endpoint: " + endpointName);
    }
} catch (InterruptedException e) {
    Thread.currentThread().interrupt();
    log.error("delay interrupted: " + e.getMessage());
}

Also in your application, increase theretry wait

<entry key="retryWait" value="60"/>
6

@Venu9000 Sleeping thread increases the aggregation time. Also, the latest is 401, not 429. Your application is not partitioned, possibly as it is a shared limit, you are hitting this limit while other services are also utilizing it. You might want to adjust the aggregaiton timings or introduce delta aggregation. that should help you out.

7

“I am frequently encountering a 429 error. When I increased the thread sleep time to 2000ms from 200ms, I started receiving a 401 error instead. Could you please advise on how I can implement delta aggregation in this situation?”

8

did you update the retry limit to 60 as well??

looking at the 401 error after increasing sleep time, this is a token expiration issue — your OAuth/Basic token is expiring during the long aggregation run because the 2000ms sleep is making the total runtime long enough that the session times out mid-aggregation.

can you share your before operation rule??

9

did you update the retry limit to 60 as well?? yes updated to 90 also.

image
image742×492 11.7 KB

before operation rule code shared via chat. Could you please check.

10

You are calling getBearerToken(application) once at the start of the rule, but with 2000ms sleep multiplied across hundreds of users and 3 child calls each, the token expires mid-run and subsequent calls get 401.

I have fixed your rule and asked you to check the TTL from blackline, can you please check, and update it in the code, before you try. It should work post doing that

11

As discussed in the chat, still the same error : Exception during aggregation. Reason: java.lang.RuntimeException: sailpoint.connector.ConnectorException: Url: https://sbus.api.blackline.com/v1/users/141/entities, Message: 401 : Unauthorized : { “detail”: “Authentication failed due to invalid authentication credentials.”, “status”: 401, “title”: “Unauthorized” } , HTTP Error Code: 401

12

@Venu9000 How many users you have in blackline app? and what is the average execution time?

13

How many users you have in blackline app? → around 2500 Users.

and what is the average execution time? → More than 1 hr

14

This is weird. for 2500 users it should complete quicker. You might want to review your network to figure out why is it taking longer?
Also, explore delta aggregaiton option if Blackline supports. Can you share the API docs for Blackline.

15

Thank you @naveenkumar3 for you help. I am able to resolve the issue after updating the before operation rule as u suggested. Thank you one more time :slight_smile:

TR_BeforeOperationRule_BlacklineTR_206736_Generic (4).xml (3.6 KB)

17

In my tenant,

Total Number of Accounts: 900

Aggregation Time : 6 Mins

I am seeing failed account aggregations once in a while during scheduled aggregation and when we run a manual aggregation, the aggregation runs without any error. Do I still need to update the retry attributes in source config? Any suggestions please.