Best practice: two-step aggregation for Web Services connector (list IDs → detail call)

Identity Security Cloud (ISC) ISC Discussion and Questions
, , ,
1

We are designing a standard Web Services connector for a target system where we need to aggregate all user accounts into ISC.

The target system exposes a REST API, but it cannot return in a single call the full flat list of all accounts together with the complete entitlement detail for each of them (the payload would be too large / the source doesn’t support that shape). Instead, the source can only expose:

  1. A list endpoint that returns a flat list of account keys (User IDs only, no detail).
  2. A “get single account” endpoint that, given one User ID, returns the full account detail (all attributes and entitlements/authorizations for that person).

Before finalizing the API contract we are asking the source-system team to expose, we want to validate with SailPoint whether this two-call cascade pattern is a supported and recommended approach for the Web Services connector’s account aggregation, or whether we should instead build a custom connector (SaaS Connectivity SDK) to implement this logic.

Example payloads

Call 1 – List Accounts (flat list of keys only)

{
  "accounts": [
    { "userId": "AC10001" },
    { "userId": "AC10002" },
    { "userId": "AC10003" }
  ],
  "nextPageToken": "eyJvZmZzZXQiOjEwMH0="
}

Call 2 – Get Single Account (full detail, called once per userId returned by Call 1)

{
  "userId": "AC10001",
  "firstName": "Mario",
  "lastName": "Rossi",
  "status": "ACTIVE",
  "department": "IT Security",
  "entitlements": [
    { "id": "ROLE_ADMIN", "type": "role" },
    { "id": "APP_ACCESS_CRM", "type": "application" },
    { "id": "GROUP_FINANCE_RO", "type": "group" }
  ]
}

What we would like to understand

  1. Is it considered best practice (or even supported out of the box) for a standard Web Services connector to build full account aggregation by:
    • first paging through a “list” endpoint to collect all account keys, and
    • then issuing one “Get Single Object” call per key to enrich each account with full entitlement detail?
  2. Historically the “Get Single Account” call in the Web Services connector framework is used for delta/provisioning confirmation scenarios, not as part of the main aggregation loop. Can it legitimately be reused inside the aggregation task itself, or is that outside its intended design?
  3. Are there known performance/rate-limit implications of this “N+1 calls” pattern (1 list call + N single-account calls) during a full aggregation, especially for larger populations (tens of thousands of accounts)? Any guidance on throttling, pagination size, or parallelization supported by the connector framework?
  4. If this pattern is not well supported by the standard Web Services connector, would you recommend implementing it instead via the SaaS Connectivity SDK (custom connector), where we have full control over the aggregation loop logic?
  5. Are there any reference implementations, community connectors, or documentation examples that follow this same “list then enrich” cascade pattern we could use as a model?

Any guidance, documentation pointers, or examples of similar implementations would be greatly appreciated.

Thank you in advance for your help.

2

Hi @ffalcitelli,

Great questions! Here are some thoughts based on my experience:

While I wouldn’t call it a “best practice,” it is definitely supported out of the box and doable. However, the configuration is slightly different from what you described. Instead of using a “Get Single Account” operation in the loop, you would define two Account Aggregation operations and use the Parent Endpoint feature:

Account Aggregation 1 (Parent): Calls the endpoint to list all user IDs.
Response Mapping: userId → userId

Account Aggregation 2 (Child): Fetches the full account details.
Example Context URL: /users/$response.userId$
Configuration: Set the Parent Endpoint to Account Aggregation 1
Map all your final attributes here.

The primary concern with this approach is scalability (the N+1 problem). If you have over 100k users, making a separate API call for each individual user could lead to extremely long aggregation times or trigger rate limits on the target API. For a smaller population (e.g., under 500 users), this usually isn’t a problem.

You are correct, using the “Get Single Account” operation within the aggregation loop is outside its intended design. It should only be used for the scenarios you mentioned (delta/provisioning confirmation). To achieve the iteration you want during aggregation, you should use the Parent Endpoint pattern with multiple Account Aggregation operations, as outlined above.

Yes, absolutely (as mentioned in point 1). The specific implications depend entirely on the target API and its limits. Since the API requires requesting full user information one user at a time, your options for optimizing are limited. Standard throttling, pagination, and parallelization features generally rely on the API’s ability to return multiple users in a single call or to filter efficiently.

Since the Web Services connector does support this via Parent Endpoints, a custom connector might not be necessary. More importantly, the SaaS Connectivity SDK won’t solve the underlying N+1 issue, your custom code would still need to make the exact same number of API calls and would face the identical performance and rate-limiting bottlenecks. The bottleneck here is the target API’s design, not the SailPoint connector.

The official documentation covers the Parent Endpoint configuration logic I mentioned in point 1. It’s supported OOTB, and I have successfully used this pattern many times in the past.

You can find the documentation here: JSON Aggregation with Parent Endpoints

3

Thanks Tyler,

In our production scenario we have around 10,000 users. I understand that a full aggregation with this two-call cascade pattern (list IDs + get single account per user) could take a while to complete.

Would this be a problem on your end, performance- or timeout-wise?

Also, is there a “Plan B” we should consider? The client is not able to expose a single getAllAccounts API that returns full account details in one call (or even paginated) — the “list IDs + get single account” cascade is the only option they can offer for now.

Thanks again for your help.

4

It definitely sounds like utilizing the Parent Endpoint pattern (the “list then enrich” cascade) is your best and likely only path forward given the constraints of the target API.

To give you an idea of the performance implications, we can do some quick math. Since you will be making an “N+1” number of calls (1 call to get the list, plus N calls for each individual user’s details), the total aggregation time will heavily depend on the target API’s response latency.

For example, let’s assume a population of 10,000 users (meaning 10,001 API calls):

Fast API (~100ms response time): 10,000 calls * 0.1 seconds = 1,000 seconds (approx. 16.6 minutes)
Average API (~500ms response time): 10,000 calls * 0.5 seconds = 5,000 seconds (approx. 1 hour 23 minutes)
Slow API (~1 second response time): 10,000 calls * 1.0 seconds = 10,000 seconds (approx. 2.7 hours)
If your user base scales up to 50,000 users, a 500ms API response time would push your aggregation time to roughly 7 hours.

Any performance issues or timeout errors you encounter will be dependent on the target API’s behavior (such as rate limits and response times) rather than an issue on the SailPoint side.

The only true way to know how it will behave in your specific environment is to test it. Thankfully, implementing the Parent Endpoint configuration is pretty straightforward in the SailPoint UI, so it shouldn’t take too much effort to set up a quick test to measure those actual API response times.

Let me know if you have any issues/concerns while testing!

5

Hi @ffalcitelli ,

Given the constraints you mentioned, no pagination and no possibility of including the access details in the list accounts endpoint, the Parent Endpoint approach seems to be the most suitable option with the standard Web Services connector.

I recently worked on a similar case with around 5K accounts. We loaded 250 accounts per page and then made 250 individual calls to retrieve the additional details, resulting in around 20 list calls + 5K detail calls in total. We did sometimes encounter 5xx errors due to the request volume.

For your 10K users, I would therefore check with the API team whether the server can reliably handle this volume of requests and whether there are any rate limits or throttling/concurrency restrictions.

Another option would be After Operations Rules, but this would not reduce the number of API calls and would likely have a similar performance impact.