What's the best approach to retrieve a large dataset from ISC using TypeScript SDK?

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

I need to retrieve a large dataset from ISC using the TypeScript SDK (v2025), for example, all accounts or identities.

Because the dataset can be very large, I want to avoid loading all results into memory at once. Instead, I’d like to fetch the data in batches and process each batch incrementally.

My initial approach was to use limit + offset pagination, for example:

const response = await searchV2025Api.searchPost({
  searchV2025: searchParams,
  limit,
  offset,
});

However, I noticed a comment in the REST API documentation stating that for large datasets (over ~10K records), searchAfter paging should be used instead.

I also saw that the TS SDK provides a built-in paginator, but I’m concerned that it may load the entire result set into memory, which I need to avoid.

What is the recommended way to efficiently iterate over large result sets in batches using the TS SDK (v2025) without holding all the data in memory?

2

Hi @liza_s,

I appreciate this question.

I benchmarked the approaches and here is what I found with 10,000 events.
FYI: I used events as I only had less than 500 identities.

Method Time and Peak memory
SDK Paginator 12.02s / 67.87 MB
searchAfter 6.36s / 66.67 MB

For datasets under 10K: SDK’s Paginator.paginateSearchApi() is simple and fast.
For datasets larger than 10K: Use searchAfter with smaller batch size to balance of speed and memory.

:slight_smile:
Thanks

3

It is simple to add the parameter count=true to your request. The response headers will then include a parameter called X-Total-Count, which specifies the total number of entries.

Based on X-Total-Count, continue iterating using limit and offset.

Example:

  • First call: offset=0&limit=500&count=true

  • Second call: offset=500&limit=500&count=true

  • Third call: offset=1000&limit=500&count=true

Continue until the offset reaches the value of X-Total-Count.

image
image1318×844 66.6 KB

4

Hi, thank you for your answer!

Do you know if there’s a fundamental difference between using Paginator and paginating by myself using limit and offset?

I would like to avoid loading all the data in memory (as the paginator does).

5

In terms of functionality, the way Paginator from the SDK internally uses the Search API if you look at the TypeScript SDK. However, there’s an important distinction: the SDK Paginator uses offset-based pagination, which may be subject to the 10,000 record limit for Search API. If you need to retrieve more than 10,000 records, you’ll need to use the Search API directly with searchAfter pagination.

In terms of memory, Paginator makes all the API calls necessary, consolidate the responses and finally returns. If you want to optimize memory by processing each batch immediately, you can call the Search API directly in a loop.
:slight_smile:
Thanks,
Amar

6

Hi @liza_s,

A request I have is to mark the best answer as solution. If you believe there is a better solution, please do share and mark that response as solution.
:slight_smile:
Thanks,
Amar