Issue with Pulling Full Identities List from Postman

AlexaNicolas · December 2, 2024, 3:12pm

I am encountering an issue when attempting to retrieve the complete list of identities (over 60,000) using Postman. There seems to be an offset or limitation problem that prevents pulling more than 10,000 identities “AT ONCE”.

Details of the Issue:

When I set the offset to 9,500 and the limit exceeds 500, I receive a 400 Bad Request.
If the offset is set to 9,000 and the limit to 1,000, I receive a 200 response with valid data. However, if I adjust the offset to 9,001 and maintain the limit at 1,000, I still get a 400 Bad Request.

Screenshots:

Request for Assistance:

Is there a known resolution for this limitation to successfully retrieve all identities with 1 query via Postman. Any guidance would be greatly appreciated.

Thank you.

mpotti · December 2, 2024, 8:59pm

You are correct there is a hard limit of the number of identities you can pull back from a single API call. This design is to ensure that the system does not become over loaded.

To create a report like this I would use a tool like PowerShell to send multiple api calls and paging through until the end.

For limits please see:

Here is an example how I would do it in powershell.

  $OffSet = 0
    $AccountUrl = "https://$($Tenant).api.identitynow.com/v3/accounts?offset=$($OffSet.ToString())&limit=250&count=true&filters=sourceId eq `"$($SourceID)`" and identity.identityState eq `"active`" and uncorrelated eq false and identity.identityState eq `"ACTIVE`""
    $Accounts = @()
    $Headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $Headers.Add("Content-Type", "application/json")
    $Headers.Add("Accept", "application/json")
    $Headers.Add("Authorization", "$($BearerToken.token_type) $($BearerToken.access_token)")

    $AccountResponse = Invoke-WebRequest -Uri $AccountUrl -Headers $Headers -Method Get

    [int32]$TotalRecords = $AccountResponse.Headers.'X-Total-Count'[0]
    $Accounts += $AccountResponse.Content | ConvertFrom-Json -Depth 100

    $FullEpicInfo = @()

    while($OffSet -le $TotalRecords){
        
        $Offset = $OffSet + 250
        $AccountUrl = "https://$($Tenant).api.identitynow.com/v3/accounts?offset=$($OffSet.ToString())&limit=250&count=true&filters=sourceId eq `"$($SourceID)`" and identity.identityState eq `"active`" and uncorrelated eq false and identity.identityState eq `"ACTIVE`""
        $AccountResponse = Invoke-WebRequest -Uri $AccountUrl -Headers $Headers -Method Get
        $Accounts += $AccountResponse.Content | ConvertFrom-Json -Depth 100
    }

sk8er23 · December 2, 2024, 9:23pm

There is a hard limit, this may be useful

"The search API in Identity Security Cloud leverages Elasticsearch functionality, which returns a maximum of 10,000 records by default. However, you can page more than 10,000 records by using the “searchAfter” property.

The searchAfter capability provides the ability to page on sorted field values, instead of offset paging. For example, if you sort by ID and page 100 records at a time, you can take the 1st page of 100 records, pass the last ID from that record set into your next search, and the next search will return the next 100 records after that ID. You continue that pattern of using the last value passed into searchAfter until the end of the result set. This allows you to page past the 10,000 record limit until you reach the final record."