Webservice connector issue

IdentityIQ (IIQ) IIQ Discussion and Questions
,
1

Hi I’m using webservice connector to onboard an application and the response has 27k elements and to process them it is taking more then 5 hrs. Is there any chance we can reduce the processing time ??

This is the afterrule im using

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathFactory;
import java.util.HashMap;
import java.util.Map;
import java.util.ArrayList;
import java.io.ByteArrayInputStream;

Log log = LogFactory.getLog("sailPoint.connector.webservice.AfterRule");

Map updatedMapInfo = new HashMap(); // Initialize empty map to store parsed response

if (rawResponseObject != null) {
    // Clean the response to handle special characters
    String cleanedResponse = rawResponseObject.replace("", "a")
                                             .replace("&", "a")
                                             .replace("
", "a");

    try {
        // Initialize the XML parser
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        factory.setNamespaceAware(true);
        DocumentBuilder builder = factory.newDocumentBuilder();
        
        // Parse the cleaned XML response
        Document document = builder.parse(new ByteArrayInputStream(cleanedResponse.getBytes("UTF-8")));

        // Set up XPath for querying the XML document
        XPathFactory xpathFactory = XPathFactory.newInstance();
        XPath xpath = xpathFactory.newXPath();

        // XPath expression to find all <Lookup> elements
        XPathExpression lookupExpr = xpath.compile("//*[local-name()='Lookup']");
        NodeList lookupNodes = (NodeList) lookupExpr.evaluate(document, XPathConstants.NODESET);

        ArrayList<Map> processedResponseList = new ArrayList();
        ArrayList storedCodes = new ArrayList();  // To store extracted 'Code' values

        // Process only the first 100 <Lookup> elements
        int maxLookups = Math.min(lookupNodes.getLength(), 10000);
        for (int i = 0; i < maxLookups; i++) {
            Node lookupNode = lookupNodes.item(i);
            if (lookupNode != null) {
                // Extract relevant fields using XPath
                String id = xpath.evaluate(".//*[local-name()='ID']", lookupNode);
                String code = xpath.evaluate(".//*[local-name()='Code']", lookupNode);

                // Create a map to store the extracted fields for the current <Lookup> element
                Map processedResponseObject = new HashMap();
                processedResponseObject.put("ID", id);
                processedResponseObject.put("User Name", code);

                // Add the 'Code' to the list of stored codes
                if (code != null && !code.isEmpty()) {
                    storedCodes.add(code);
                }

                // Add the current parsed object to the list
                processedResponseList.add(processedResponseObject);
            }
        }

        // Log and return the parsed response
        log.debug("Parsed Response List (First 10 Lookups): " + processedResponseList);

        // Store the processed response into a map to return it
        updatedMapInfo.put("data", processedResponseList);  // This key might vary based on the endpoint's expected structure

        // Store the 'Codes' in the connectorStateMap for later use
        Map connectorStateMap = new HashMap();
        connectorStateMap.put("storedCodes", storedCodes);  // Storing in connectorStateMap

        log.debug("Stored Codes for Later Use in connectorStateMap: " + storedCodes);

        return updatedMapInfo;  // Return the parsed response

    } catch (Exception e) {
        // Log any errors during parsing
        log.error("Error parsing cleaned XML response: " + cleanedResponse, e);
    }
} else {
    log.warn("Raw Response Object is null. No processing will be done.");
}

// If the response is null or an error occurred, return an empty map
return updatedMapInfo;
2

Hi @pradeepireddy_123,

the execution time depends directly on the number of objects read, number of attribute for each object(and if you apply some rules) and server\network resources.

  • N.Object: read only necessary, you can filter for only active account, even if the requierement permit to you
  • N.Attribute: read only necessary
  • Resources: you can use partitioning and thread setup
3

@pradeepireddy_123
The main bottleneck in your script is the repeated use of XPath evaluations within the loop over a potentially large number of <Lookup> elements. Each XPath evaluation can be expensive, especially when dealing with thousands of nodes. To optimize the script and reduce processing time, you can:

  1. Avoid XPath Evaluations Within the Loop: Instead of using XPath to extract the ID and Code elements for each <Lookup> node, navigate the DOM tree directly.
  2. Pre-compile XPath Expressions: If you must use XPath, compile the expressions once outside the loop and reuse them.
  3. Optimize XPath Expressions: Make the XPath expressions more specific to reduce evaluation time.

Check if the below code helps -

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import java.util.HashMap;
import java.util.Map;
import java.util.ArrayList;
import java.io.ByteArrayInputStream;

Log log = LogFactory.getLog("sailPoint.connector.webservice.AfterRule");

Map updatedMapInfo = new HashMap(); // Initialize empty map to store parsed response

if (rawResponseObject != null) {
    // Clean the response to handle special characters
    String cleanedResponse = rawResponseObject.replace("&#xE;", "a")
                                             .replace("&amp;", "a")
                                             .replace("&#xD;", "a");

    try {
        // Initialize the XML parser
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        factory.setNamespaceAware(true);  // Important for handling namespaces
        DocumentBuilder builder = factory.newDocumentBuilder();

        // Parse the cleaned XML response
        Document document = builder.parse(new ByteArrayInputStream(cleanedResponse.getBytes("UTF-8")));

        // Find all <Lookup> elements
        NodeList lookupNodes = document.getElementsByTagNameNS("*", "Lookup");

        ArrayList<Map> processedResponseList = new ArrayList();
        ArrayList storedCodes = new ArrayList();  // To store extracted 'Code' values

        // Process up to 10,000 <Lookup> elements
        int maxLookups = Math.min(lookupNodes.getLength(), 10000);
        for (int i = 0; i < maxLookups; i++) {
            Node lookupNode = lookupNodes.item(i);
            if (lookupNode != null) {
                // Extract 'ID' and 'Code' by navigating the DOM directly
                String id = null;
                String code = null;

                NodeList childNodes = lookupNode.getChildNodes();
                for (int j = 0; j < childNodes.getLength(); j++) {
                    Node child = childNodes.item(j);
                    if (child.getNodeType() == Node.ELEMENT_NODE) {
                        String localName = child.getLocalName();
                        if ("ID".equals(localName)) {
                            id = child.getTextContent();
                        } else if ("Code".equals(localName)) {
                            code = child.getTextContent();
                        }
                    }
                    // Break early if both values are found
                    if (id != null && code != null) {
                        break;
                    }
                }

                // Create a map to store the extracted fields for the current <Lookup> element
                Map processedResponseObject = new HashMap();
                processedResponseObject.put("ID", id);
                processedResponseObject.put("User Name", code);

                // Add the 'Code' to the list of stored codes
                if (code != null && !code.isEmpty()) {
                    storedCodes.add(code);
                }

                // Add the current parsed object to the list
                processedResponseList.add(processedResponseObject);
            }
        }

        // Log and return the parsed response
        log.debug("Parsed Response List (First 10 Lookups): " + processedResponseList.subList(0, Math.min(10, processedResponseList.size())));

        // Store the processed response into a map to return it
        updatedMapInfo.put("data", processedResponseList);  // This key might vary based on the endpoint's expected structure

        // Store the 'Codes' in the connectorStateMap for later use
        Map connectorStateMap = new HashMap();
        connectorStateMap.put("storedCodes", storedCodes);  // Storing in connectorStateMap

        log.debug("Stored Codes for Later Use in connectorStateMap: " + storedCodes);

        return updatedMapInfo;  // Return the parsed response

    } catch (Exception e) {
        // Log any errors during parsing
        log.error("Error parsing cleaned XML response: " + cleanedResponse, e);
    }
} else {
    log.warn("Raw Response Object is null. No processing will be done.");
}

// If the response is null or an error occurred, return an empty map
return updatedMapInfo;
5

@pradeepireddy_123 - That’s perfect. Please mark the query as solved and raise a new topic discussion regarding the new issue.

Thank you