How to implement rate limit in webservice connector rules

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

Hi team,

We’ve integrated one of the application using webservice out of the box connector. Currently during aggregation, we started seeing 503 service unavailable error. The application has limited the integration 5/10 (RPS/Burst) limits. How can we handle this on SailPoint side. Anyone has done this type of similar implementation.

Even this is happening during attribute sync as well.

Thanks in advance for the inputs!

Please consider addressing the following when creating your topic:

  • What have you tried?
  • What errors did you face (share screenshots)?
  • Share the details of your efforts (code / search query, workflow json etc.)?
  • What is the result you are getting and what were you expecting?
2

503Errr
503Errr601×267 5.76 KB

3

Are you able to increase the limit in your use case? One thing I have noticed with NERM using the WebServices connector is that if the page ends on last record on the page it will error out. Using your screen print say the page ended on record 6049 as there are not more records the server errors with a similar error message. Fix has been add another record.

4

How are you making your web services calls? Did you configure them through the UI or are you using a rule?

If you use a rule, you can control how the data is retrieved and how it is processed.

5

@Shonnegowda have you tried this in postman ? Also you can check with App team if this rate can be increased

6

I’m using webservice before operation rule for the update account operation for the attribute sync, where in I’ve to fetch the user mandatory attributes by using Get User API calls to update the identity attributes and also handling to update the additional attributes using another API calls. For aggregation also I’m using the rule

7

This is nothing to do with the pagination limit. This is more of request per second, which we need to configure at our end. The application team configuration handled 5/10 RPS for this integration, they can able to send the proper error message to distingusih if required.

8 
import java.util.Map;
import java.util.List;
import java.util.Collections;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import connector.common.JsonUtil;

Log logger = LogFactory.getLog("logs.AddEntitlement");

if (logger.isTraceEnabled()) logger.trace("Enter AddEntitlement");

// Initialize a map to hold updated information
Map updatedMapInfo = new HashMap();

if (logger.isTraceEnabled()) logger.trace("In AddEntitlement processedResponseObject: " + processedResponseObject);

String baseURL = (String) application.getAttributeValue("genericWebServiceBaseUrl");
String prefix = "In AddEntitlement after operation ";

if (null == requestEndPoint) {
    if (logger.isTraceEnabled()) logger.trace(prefix + " requestEndPoint is null, aborting operation.");
    return;
}

String accessToken = requestEndPoint.getHeader().get("Authorization");

// to store fetched user attributes
Map userAttributesMap = new HashMap();

Map fetchUserAttributesDetails(String prefix, String accessToken, String baseURL, String nativeIdentity) {
    Map userAttributeDetails = new HashMap();
    String getUserAttributesURL = baseURL + "/identities/" + nativeIdentity + "/attributes";
    if (logger.isTraceEnabled()) logger.trace(prefix + "Fetching user attribute details from URL: " + getUserAttributesURL);

    try {
        Map headers = new HashMap();
        headers.put("Authorization", accessToken);

        List allowedStatusCodes = Collections.singletonList("2**");
        String response = restClient.executeGet(getUserAttributesURL, headers, allowedStatusCodes);
        Map responseMap = JsonUtil.toMap(response);

        if (responseMap != null) {
            if (logger.isTraceEnabled()) logger.trace(prefix + "Response from API: " + responseMap);
            List attributes = (List) responseMap.get("attributes");

            if (attributes != null) {
                for (Map attribute : attributes) {
                    String key = (String) attribute.get("key");
                    List valueList = (List) attribute.get("value");

                    if (valueList != null && !valueList.isEmpty()) {
                        String value = (String) valueList.get(0);
                        userAttributeDetails.put(key, value);
                    }
                }
            } else {
                if (logger.isTraceEnabled()) logger.trace(prefix + " No attributes found in the response.");
            }
        }
    } catch (Exception e) {
        if (logger.isTraceEnabled()) logger.trace(prefix + " Error fetching user details: " + e.getMessage(), e);
    }
    return userAttributeDetails;
}

if (processedResponseObject != null) {
    for (Map accountAttributes : processedResponseObject) {
        if (accountAttributes != null) {
            // Add static dummy entitlement value
            accountAttributes.put("entitlement", "dummy_ROLE");
            if (logger.isTraceEnabled()) logger.trace("processedResponseObjectMap after entitlement added: " + accountAttributes);

            String nativeIdentity = (String) accountAttributes.get("cwid");

            if (nativeIdentity != null) {
                // Check cache first
                if (!userAttributesMap.containsKey(nativeIdentity)) {
                    Map userAttributes = fetchUserAttributesDetails(prefix, accessToken, baseURL, nativeIdentity);
                    userAttributesMap.put(nativeIdentity, userAttributes);
                }
                accountAttributes.putAll(userAttributesMap.get(nativeIdentity));
            }

            if (logger.isTraceEnabled()) logger.trace("processedResponseObjectMap after user attributes merged: " + accountAttributes);
        }

        try {
        long startTime = System.currentTimeMillis();
        long delay = 1000; // 1 second delay
        while (System.currentTimeMillis() - startTime < delay) {
            if (logger.isTraceEnabled()) logger.trace("Waiting for " + (delay - (System.currentTimeMillis() - startTime)) + "ms before upsert.");
        }
        } catch (InterruptedException e) {
            if (logger.isTraceEnabled()) logger.trace(prefix + " Error adding delay " + e.getMessage(), e);
        }
        
    }
    // Put the updated response object into the updatedMapInfo
    updatedMapInfo.put("data", processedResponseObject);
}

if (logger.isTraceEnabled()) logger.trace("Exit AddEntitlement " + updatedMapInfo);
return updatedMapInfo;
9

Just like you have rule for updating attributes, you can have a rule for aggregation. See how you have put a delay in for making the updates, do the same thing when you are aggregating. Authenticate to the source, start the data collection, get a first set of data, delay, get the next set of data, delay, get the next set of data.

You do not have to have anything configured on the webservices aggregation section in ISC. Just write a rule and attach it to the aggregation.

10

The rule I’ve uploaded here it is for aggregation only, but the delay not make sure about the 5 request per second

11

I’ve attempted to use Thread.Sleep(), but its not allowing to save the file and warning to remove this.

‘idn://***.identitynow.com/beta/connector-rule-script/7337d2f73b9842139659697519acdf1a/Rule-WebServiceAfterOperation-PACE-Aggregation’ (Error: Illegal value “[{“line”:90,“column”:29,“message”:“Remove reference to Thread”}]” for field “sourceCode.script”.)

12

Thread.sleep() is specifically not allowed in rules. You may find another function or just use a loop like you started above.

13

That wait is not helping out. Need to implement something externally

14

Is anyone used retry mechanism based on HTTP response code, which is getting from an application API’s

15

Is the application returning the http rate limit headers?

Are you able to modify the application?

16

Currently it is not. Application team has made some changes at their end. We’ll test it. Thanks!

17

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.