Full Source Aggregation

Problem

We schedule aggregations to automatically load new data on the source into Identity Now on a regular basis. If the source supports delta aggregation, we prefer using that option to lower the aggregation times and minimize load on the system. But, there are times when the system demands a full source aggregation to be run to pick up certain changes else the account in IDN remains unchanged. Example: OU Moves or Account deletions are only processed during full source aggregations.

When delta aggregation is Enabled in Source configuration, all the scheduled aggregation by default will run as deltas, there is no option to specify an additional full aggregation run on a different schedule.

Solution

To handle this problem, I came up with a workflow that will trigger an additional full source aggregation on schedule.

Configuration Steps

  • Create a Personal Access Token (PAT) to call the Identity Now Rest APIs from the workflow.
  • Create a Workflow with a Scheduled Trigger
  • Create a HTTP Request Action to Disable Delta Aggregation in Source Configuration
  • Create a HTTP Request Action to Run the Full Source Aggregation (Optimization flag optional)
  • Create n HTTP Request Action to Enable Delta Aggregation in Source Configuration

Workflow

JSON

Here is the workflow script you can use. Please update the HTTP steps to your environment URL and credentials, also update the Source IDs and aggregation schedule trigger as per your needs. You can disable optimization in the HTTP request action for full aggregation if required.

FullSourceAggregation20231108.json (3.9 KB)

8 Likes

Excellent !! This will be very helpful

3 Likes

I love your efforts @sharvari and the content is detailed as always.

Just an addition, incase if tenant doesn’t have workflow license then
I guess we can create a PowerShell or Python script and schedule it in IQ Service which will run API calls for full aggregation or whatever the operations we need in a sequence.

We have sequential task launcher in IIQ but not in IDN, guess we can build the same in your workflow or in scripting as i mentioned.

1 Like

Yes, we used powershell to accomplish this before getting workflows in IDN.

Another thing that will have to be modified in this solution is the replacement of /cc/loadAccounts API with its equivalent v3 or beta API once it’s available.

1 Like

Not sure if you have observed this. OU changes does not require full aggregation anymore. SailPoint released a fix in AD connector in August 14, 2023.
https://community.sailpoint.com/t5/SaaS-Release-Notes/tkb-p/saas-release-notes?date=2023-08-14&env=production
According to this the Active Directory connector now instantly returns the Resource Object to IdentityNow on any OU changes done by the AC_New Parent, which can be further utilized to any rule to work with the updated Resource Object data values.

But good to have this option for other sources.

I’m just chucking a comment in here that I found and used this today in ISC. Thank you! I needed to do this for our Okta source. In case anyone else comes upon this use-case and is new to Sailpoint like me: The loadAccounts API isn’t advertised in the main API suite (is some super-secret one?) and uses the source’s cloudExternalId (can be viewed from vscode or API I imagine) vs. the usual SourceID.

Yes, it’s one of the older cc API’s, you won’t find them listed on the documentation. It is about to be deprecated any day now – more details here.

If you are using this workflow, please use the below beta API for account aggregation instead of cc loadAccounts.