Using WebService connector rules to streamline and accommodate “interesting” sources

Description

Continuing the conversation of connectivity, Stephen will provide you with a walkthrough of using WebService Before and After Operation connector rules to aggregate data when the provider’s data is in a state that is hard or impossible to tie to the Account itself, as well as to cache data from aggregation start to aggregation end to reduce network use and avoid API rate limits.

API Layout

each of my API endpoints can be used as bulk endpoints, or singular get-by-ID.

each Employee is in the form of:

{
  "id": 1,
  "firstName": "Rutter",
  "lastName": "Mailey",
  "emailAddress": "[email protected]",
  "companyId": 3,
  "departmentId": 15,
  "WorkType": 2,
  "jobId": 10
}

and each Job, Company, and Department are in the form of:

{
  "id": "2",
  "title": "Batman"
}

Additional Resources

Web Service After Operation Rule

the source code below can be applied as a Web Services After Operation Rule to:

  • use temporary storage (transientValues) data if present
  • call additional APIs and parse/store their values
    • the RestClient operates in the context of the connector itself, inheriting authentication, headers, etc. there is no need to hard-code any credentials or manually handle authentication
  • iterating over each User from the paginated data, append additional data to the response, such as a job title
import org.json.*;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import connector.common.JsonUtil;
import connector.common.JsonUtil.*;
import sailpoint.object.*;
import connector.common.Util;

/*
 * CREATED BY: Stephen Holinaty
 * Date: 4/1/2024
 * Purpose: Demo code for Developer Days 2024, presenting how Web Service After Operation 
 *          rules can be used for efficiently calling, and using temporary storage to 
 *          utilize, additional APIs to overload the standard output with additional data
 */

 
//setting a log prefix allows us to easily filter ccg.log for relevant items
String logPrefix = "WSAO_DevDays2024";

//retrieve the transient values, if they exits, from application storage. will be Null if not set.
Map transientValues = (Map) application.getAttributeValue("transientValues");

//build our return object structure
List returnObj = new ArrayList();
Map processedData = new HashMap();
log.info(logPrefix + " starting to process");

//only call additional APIs if we do not already have them in memory
if(transientValues == null) {
    ///////// START LOOKUPS ///////////
    Map transientValues = new HashMap();
    
    //set up the requirements for making API calls
    List<String> allowedStatuses = new ArrayList();
    allowedStatuses.add( "2**" );
    Map header = requestEndPoint.getHeader();

    //retrieve the configured Base URL from the Source
    String baseUrl = application.getAttributeValue("genericWebServiceBaseUrl");

    //call the jobs API, build a key/value Map of job IT : job Title
    Map JobCodeLookupMap = new HashMap();
    try{
        String jobsurl = baseUrl + "/jobs";
        //executeGet operates in the context of the source restClient.  
        //will follow Authentication on the source/endpoint.
        String jobs = restClient.executeGet(jobsurl, header, allowedStatuses );
        JSONArray jobsJSONArray = new JSONArray(jobs);
        for (int i = 0; i < jobsJSONArray.length(); i++) {
            JSONObject jobJSONObject = jobsJSONArray.getJSONObject(i);
            String jobId = jobJSONObject.getString("id");
            String jobTitle = jobJSONObject.getString("title");
            JobCodeLookupMap.put(jobId,jobTitle);          
        }  
        //store the resultant map into transient values
        transientValues.put("JobCodeLookupMap", JobCodeLookupMap);
    } catch (Exception e) {
        log.error(logPrefix + "********ERROR AFTER RULE: " + e.getMessage(), e );
    }

    //repeat for the company API
    Map companyLookupMap = new HashMap();
    try{  
        String companyUrl = baseUrl + "/company";
        String companies = restClient.executeGet(companyUrl, header, allowedStatuses );
        JSONArray companyJSONArray = new JSONArray(companies);
        for (int i = 0; i < companyJSONArray.length(); i++) {
            JSONObject companyJSONObject = companyJSONArray.getJSONObject(i);
            String companyId = companyJSONObject.getString("id");
            String companyName = companyJSONObject.getString("companyName");
            companyLookupMap.put(companyId,companyName);          
        }
        transientValues.put("companyLookupMap", companyLookupMap);
    } catch (Exception e) {
        log.error(logPrefix + "********ERROR AFTER RULE: " + e.getMessage(), e );
    }

    //repeat for the department API
    Map departmentLookupMap = new HashMap();
    try{
        String departmenturl = baseUrl + "/department";
        String departments = restClient.executeGet(departmenturl, header, allowedStatuses );
        JSONArray departmentJSONArray = new JSONArray(departments);
        for (int i = 0; i < departmentJSONArray.length(); i++) {
            JSONObject departmentJSONObject = departmentJSONArray.getJSONObject(i);
            String departmentId = departmentJSONObject.getString("id");
            String departmentName = departmentJSONObject.getString("name");
            departmentLookupMap.put(departmentId,departmentName);          
        }  
        transientValues.put("departmentLookupMap", departmentLookupMap);
    }  catch (Exception e) {
        log.error(logPrefix + "********ERROR AFTER RULE: " + e.getMessage(), e );
    }

    //store "our" transient values into the application (until the aggregation completes, wherein it will erase)
    application.setAttribute("transientValues", transientValues);
    log.error(logPrefix + "setting transients");
}
    
/////// END LOOKUPS //////////

// ensure that we have data to actuall operate on
if(rawResponseObject != null && processedResponseObject != null){
    //retrieve the transientValues from application storage
    transientValues = (Map) application.getAttributeValue("transientValues");

    //iterate over every Record in the response
    // processedResponseObject will have the JSONPath values defined on the source endpoint.
    for (Map employeeEntry : processedResponseObject) {
        log.error(logPrefix + "updating processed data");

        // grab key values from the record, from the JSONPath'd output of the endpoint itself
        String jobId = employeeEntry.get("jobId");
        String companyId = employeeEntry.get("companyId");
        String departmentId = employeeEntry.get("departmentId");

        //null safety. dont look it up if it doesnt exist.
        if (transientValues.containsKey("JobCodeLookupMap")){
            HashMap JobCodeLookupMap = (HashMap) transientValues.get("JobCodeLookupMap");
            //only try to add the value if it actually exists.
            if (JobCodeLookupMap.containsKey(jobId)){
                employeeEntry.put("jobTitle",JobCodeLookupMap.get(jobId));
            } else{
                log.error(logPrefix + "lookup failed for jobcode: " + jobId);
            }
        }
        
        //repeat for Company code
        if (transientValues.containsKey("companyLookupMap")){
            HashMap companyLookupMap = (HashMap) transientValues.get("companyLookupMap");
            if (companyLookupMap.containsKey(companyId)){
                employeeEntry.put("companyName",companyLookupMap.get(companyId));
            }else{
                log.error(logPrefix + "lookup failed for company code: " + companyId);
            }
        }

        //repeat for department
        if (transientValues.containsKey("departmentLookupMap")){
            HashMap departmentLookupMap = (HashMap) transientValues.get("departmentLookupMap");
            if (departmentLookupMap.containsKey(departmentId)){
                employeeEntry.put("departmentName",departmentLookupMap.get(departmentId));
            }else{
                log.error(logPrefix + "lookup failed for department: " + departmentId);
            }
        }

            //add the modified record to our new return result
        returnObj.add(employeeEntry);
    }

    //add the data to the reply, and send it
    processedData.put("data", returnObj);
    log.info(logPrefix + "finished running");
    return processedData;
    
} else{
    log.error(logPrefix + "Response Object was empty, standard endpoint data will be passed.");
}
4 Likes

Rule and explanation have now been posted at the top level of this thread!

Thanks for the great video Stephen, always interesting to see how we can best use rules in ways that are not so well documented or straight forward to figure out.

1 Like

@tysremi Happy to help!! I wanted to appeal to a wider audience and present some “basics” being used in innovative ways, that could benefit anyone.

i should also mention that the employeeEntry items are not limited to pure “add a new field”, and can overwrite existing data as well.

anything from a simple data merge

employeeEntry.put("full name",employeeEntry.get("firstName") + " " + employeeEntry.get("lastName"));

to something more complex and overwrite existing data

employeeEntry.put("firstName", employeeEntry.get("firstName").toUpperCase()
);

or even if your API has a pesky List that you want to be a string, the util.listToCSV function can turn [“a”, “b”, “c”] into “a,b,c”

personEntry.put("FieldName", Util.listToCsv(personEntry.get("FieldName")));