Full Source Aggregation

Identity Security Cloud (ISC) ISC Community Knowledge Base
, , ,
1

Problem

We schedule aggregations to automatically load new data on the source into Identity Now on a regular basis. If the source supports delta aggregation, we prefer using that option to lower the aggregation times and minimize load on the system. But, there are times when the system demands a full source aggregation to be run to pick up certain changes else the account in IDN remains unchanged. Example: OU Moves or Account deletions are only processed during full source aggregations.

When delta aggregation is Enabled in Source configuration, all the scheduled aggregation by default will run as deltas, there is no option to specify an additional full aggregation run on a different schedule.

Solution

To handle this problem, I came up with a workflow that will trigger an additional full source aggregation on schedule.

Configuration Steps

  • Create a Personal Access Token (PAT) to call the Identity Now Rest APIs from the workflow.
  • Create a Workflow with a Scheduled Trigger
  • Create a HTTP Request Action to Disable Delta Aggregation in Source Configuration
  • Create a HTTP Request Action to Run the Full Source Aggregation (Optimization flag optional)
  • Create n HTTP Request Action to Enable Delta Aggregation in Source Configuration

Workflow

FullSourceAggregation20231106 (2)
FullSourceAggregation20231106 (2)590×1133 37.7 KB

JSON

Here is the workflow script you can use. Please update the HTTP steps to your environment URL and credentials, also update the Source IDs and aggregation schedule trigger as per your needs. You can disable optimization in the HTTP request action for full aggregation if required.

FullSourceAggregation20231108.json (3.9 KB)

12 Likes
2

Excellent !! This will be very helpful

3 Likes
3

I love your efforts @sharvari and the content is detailed as always.

Just an addition, incase if tenant doesn’t have workflow license then
I guess we can create a PowerShell or Python script and schedule it in IQ Service which will run API calls for full aggregation or whatever the operations we need in a sequence.

We have sequential task launcher in IIQ but not in IDN, guess we can build the same in your workflow or in scripting as i mentioned.

1 Like
4

Yes, we used powershell to accomplish this before getting workflows in IDN.

Another thing that will have to be modified in this solution is the replacement of /cc/loadAccounts API with its equivalent v3 or beta API once it’s available.

1 Like
5

Not sure if you have observed this. OU changes does not require full aggregation anymore. SailPoint released a fix in AD connector in August 14, 2023.
https://community.sailpoint.com/t5/SaaS-Release-Notes/tkb-p/saas-release-notes?date=2023-08-14&env=production
According to this the Active Directory connector now instantly returns the Resource Object to IdentityNow on any OU changes done by the AC_New Parent, which can be further utilized to any rule to work with the updated Resource Object data values.

But good to have this option for other sources.

6

I’m just chucking a comment in here that I found and used this today in ISC. Thank you! I needed to do this for our Okta source. In case anyone else comes upon this use-case and is new to Sailpoint like me: The loadAccounts API isn’t advertised in the main API suite (is some super-secret one?) and uses the source’s cloudExternalId (can be viewed from vscode or API I imagine) vs. the usual SourceID.

7

Yes, it’s one of the older cc API’s, you won’t find them listed on the documentation. It is about to be deprecated any day now – more details here.

If you are using this workflow, please use the below beta API for account aggregation instead of cc loadAccounts.

8

HI @sharvari,

I am trying to enable/disable Delta aggregation using API for Workday Saas connector but it’s not working although response is 200 OK.

curl --location --request PATCH 'https://sailpoint.api.identitynow.com/beta/sources/<id>' \
--header 'Content-Type: application/json-patch+json' \
--header 'Accept: application/json' \
--header 'Authorization: ••••••' \
--data '[
    {
        "op": "replace",
        "path": "/connectorAttributes/deltaAggregationEnabled",
        "value": "true"
    }
]'

Any insights on this?

Thanks.

9

Hi @nikhleshsdg,

Please check if you have setup the Delta Aggregation Events within your Workday connector. If not, please add them before executing the delta agg.

10

HI @sharvari,

Earlier Delta Aggregation Events were updated but later they were removed (not sure how). I have enabled Delta aggregation again and added events but still not able to Disable Delta aggregation through API but getting 200OK response.

Thanks.

11

Additionally, If I disable Delta Aggregation through UI and enable it again then events got removed automatically. Not sure why this is happening.

12

The disable delta agg is removing the events i guess. If yes, you need to add the events back each time using the same API call when you are enabling delta agg.

13

Yes, I can add but API is not able to Enable / Disable Delta Aggregation.

14

Have you tried executing the same APIs in Postman Or even if you do things through UI were you able to run a delta agg successfully anytime? If UI works you can monitor the API calls it is making using the Network tab in Developer tools. You can then call the same APIs from your script.

15

Thanks @sharvari, using Developer tools I got the correct payload (mentioned below) to enable/disable Delta aggregation.

[
    {
        "op": "add",
        "path": "/connectorAttributes/spConnEnableStatefulCommands",
        "value": true
    }
]

Also, if we do not add any Delta aggregation events will SailPoint aggregate all the changes?

Thanks.

16

I haven’t tried that recently, you can try and let us know your findings :slight_smile:

17

Yes sure, will update here once we have enough evidence.

Although I have noticed that sometimes API run full aggregation where all the accounts scanned but sometimes same API doesn’t scan any account.

Thanks.