Web Services source and filters in the API calls

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

We have a Web Services Source set up to pull identity data from MDStaff for credentialed providers. There are currently over 63,000 accounts in that system. The vast majority of those accounts don’t change but we are pulling all of them in everyday when we aggregate. Aggregations are taking about 8 hours to pull all of the data. The documentation says that we can filter on a last updated field but don’t know how to insert dates into the API call.

This is an example API call from the documentation.
{
“source”: “Demographic”,
“fields”: [
“FormattedNameWithDegree”,
“LastUpdated”,
“ProviderID”
],
“filter”: {
“LastUpdated”: [
{“type”: “between”, “values”: [“12/31/2021”, “01/01/2022”]},
{“type”: “in”, “values”: [null]}
]
},
“sort”: [{“LastUpdated”: “desc”}, {“FormattedNameWithDegree”: “asc”}],
“settings”: {
“IncludeArchivedProviders”: false,
“IncludeApplicants”: false
},
“resultsperpage”:10,
“page”:1
}

What i want to do instead of hard coding dates is have it do something like this

    {"type": "between", "values": ["today-2 days", "today"]},

so it’s only pulling in changes between today-2 days and today. That way we are only pulling changes from the last 2 days and if there is a failure in a API call I should get the data on the next call.

Any pointers on how to accomplish this?

2

You should be able to create an HTTP Operation of type ‘Account Delta Aggregation’ and then in Connection Settings, enable Delta Aggregation

3

Jason can you provide a little more detail?

4

Hi Scott,

See Account Delta Aggregation

You could duplicate your current account aggregation HTTP Operation, then change Operation Type to be ‘Account Delta Aggregation’ then update your GET URL (or body) to get the right date range.

Do you know the URL or exact request you need to get the delta from your endpoint source?

If you need to do something more complicated to set the URL or body for delta aggregation, then create a Web Service Before Operation rule, and attach it to this operation.