Enhancements: Updates to API Paging Limitations

Announcements Product News
, ,
43

This is my PowerShell implementation:
Github Gist

#region IDN functions
function IdentitiesSearchAfter($token, $query, $searchAfterID, $limit, $IDNdisplayNameSearch)
{
    #region IDN Search Queries
        $identitySearchqry        = "{`"query`": {`"query`": `"source.name:HR`")`"},`"indicies`": [`"identities`"],`"queryResultFilter`": {`"includes`": [`"attributes.uid`",`"attributes.displayName`",`"attributes.identificationNumber`",`"id`",`"status`",`"isManager`"]},`"sort`": [`"+id`"],`"includeNested`": false,`"searchAfter`": [`"$($searchAfterID)`"]}"
        #eventSearchQry - have multiple queries that can be used, reuse the same logic to fetch 10k+ results 
    #endregion IDN Search Queries

    $body = switch ($query) 
    {
        "identitySearch"        { $identitySearchqry }
        # "eventSearch" { $eventSearchQry } - other queries
        Default {return "#switch-invalid query"}
    }

    $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $headers.Add("Content-Type", "application/json")
    $headers.Add("Authorization", "Bearer $($IDNToken)")
    
    try   { $response = Invoke-RestMethod "https://tenant.api.identitynow.com/v3/search?count=true&offset=0&limit=$($limit)" -Method 'POST' -Headers $headers -Body $body -ResponseHeadersVariable 'searchAfter_ResponseHeader' }
    catch { Write-Error "Exception:" $_.Exception }
    
    return $searchAfter_ResponseHeader, $response
}
#endregion IDN functions

#region IdentitiesSearch
    $queryTypes = @("identitySearchqry","eventSearchQry")
    $searchAfterID = $null
    $IdentitiesSearchAfter_Header = $null
    $IdentitiesSearchAfter_Result = @()
    $IdentitiesSearchAfter_Resulttemp = $null

    foreach($queryType in $queryTypes)
    {
        #Search & Get Count of records
        $IdentitiesSearchAfter_Header, $IdentitiesSearchAfter_Resulttemp = IdentitiesSearchAfter -token $IDNToken -query $queryType -searchAfterID $searchAfterID -limit 1

        #if length >= totalcount : stop
        $verbose=$true; if($verbose) {Write-Host ">>>Initialize-1/3 IdentitiesSearch | $($queryType) | Total-Count: $([int]$IdentitiesSearchAfter_Header.("X-Total-Count")[0]) | Array Size: $($IdentitiesSearchAfter_Result.Length) | searchAfterID:$($searchAfterID)_"}
        do{   
            $verbose=$true; if($verbose) {Write-Host ">AggregateFromIDN-2/3 | $($queryType) | Total-Count: $([int]$IdentitiesSearchAfter_Header.("X-Total-Count")[0]) | Array Size: $($IdentitiesSearchAfter_Result.Length) | searchAfterID:$($searchAfterID)"}
            $IdentitiesSearchAfter_Header, $IdentitiesSearchAfter_Resulttemp = IdentitiesSearchAfter -token $IDNToken -query $queryType -searchAfterID $searchAfterID -limit 250
            $IdentitiesSearchAfter_Result += $IdentitiesSearchAfter_Resulttemp
            $searchAfterID = $IdentitiesSearchAfter_Resulttemp[$IdentitiesSearchAfter_Resulttemp.length - 1].id
            $IdentitiesSearchAfter_Resulttemp = $null
            # $IdentitiesSearchAfter_Resulttemp[$IdentitiesSearchAfter_Resulttemp.length - 1].id
        }
        while($([int]$IdentitiesSearchAfter_Result.Length) -lt $([int]$IdentitiesSearchAfter_Header.("X-Total-Count")[0]) )
        Write-Host ">>>Complete-3/3 IdentitiesSearch | $($queryType) | Total-Count: $([int]$IdentitiesSearchAfter_Header.("X-Total-Count")[0]) | Array Size: $($IdentitiesSearchAfter_Result.Length) | searchAfterID:$($searchAfterID)"
    }

    #do something with:
    $IdentitiesSearchAfter_Result
#endregion IdentitiesSearch
1 Like
44

This is cool. Any idea how I can change the Authorization to a Personal Token instead? Going to test this out now on my DEV

45

You would need to get the Access Token then put this into the same Authorization header regardless of the type of auth flow used - Personal token or client credentials, etc.

Authentication

46

Hey Rishabh,

My team and I actually started exploring updating the endpoints to the beta endpoints as you have after I posted my update. We were able to get the entitlement hash table part updated and working, but we didn’t update all of the end points. I see you have done that in your script and it is very much appreciated!

Thank you for sharing this document so quickly and for taking the time to update all of the endpoints that are referenced to also include pagination. Thanks again for sharing, this is something we will definitely utilize going forward!

47

THANK YOU! Just giving this a quick test but looks VERY promising.

1 Like
48

Can someone provide an example of how I would export more than 250 records for running a command like this one https://{tenant}.api.identitynow.com/v3/accounts?filters=sourceId in (“sourceid 1”,“sourceid 2”)

Whereas the results would return exceeding 10k rows, but for this example. What commands in Postman do I need to run for the each one to get an export of each. I havent got Powershell or anything else and need to be able to do this using Postman. Its not clear to me reading the documentation what I need to do to get more than 250

First
Second
Third…

49

I don’t think Postman is the best tool for that purpose - I could only spot a couple of examples online regarding pagination support in Postman and in most cases the recommendation is to do it in javascript/python or in general, using a more powerful tool.

Is it necessary to use the API for this report? If it’s just accounts you are after, maybe using search via UI is a better way?

In general, in order to get more than 250 results from a single endpoint, you need to query the endpoint more than once. A single query will only ever return 250 results.
In order to get more, you need to keep querying the endpoint until you have all the results you need.

If you insist on using Postman, you would need to query the endpoint once, download the results, then add an offset parameter to your call and keeping doing that until you have all the results. Bear in mind you’ll need to download each set of results and stitch them together later manually :).
This page explains pagination in IDN endpoints: Standard Collection Parameters | SailPoint Developer Community

Example:
First call:
/v3/accounts/?filters…&offset=0
Second call:
/v3/accounts/?filters…&offset=250
Third call:
/v3/accounts/?filters…&offset=500

And so on and so on.

50

@RTASB

It looks like the BETA/entitlements API is currently “broken” and not returning a cursor as expected. Curious if anyone else has run into this and assuming the code could be updated to use the offset/limit values instead to get this working again?

51

I’m using the beta/entitlements endpoint in one of my scripts with offset/limit like this

# get total number of entitlements
def get_total_entitlement_count():
    response = r.get(os.getenv("SAILPOINT_URL") + "/beta/entitlements?filters=source.id eq \"" + os.getenv("SOURCE_ID") + "\"&count=true",
                     headers=apiCallHeaders)
    print("Total number of entitlements is " + str(response.headers['X-Total-Count']))
    return int(response.headers['X-Total-Count'])


# Get list of entitlements
def get_entitlements():
    entitlement_list = []
    for offset in range(0, int(entitlement_count), 250):
        url = os.getenv("SAILPOINT_URL") + "/beta/entitlements?filters=source.id eq \"" + os.getenv(
            "SOURCE_ID") + "\"&limit=250&offset=" + str(offset)
        response = r.get(url, headers=apiCallHeaders)
        for entitlement in response.json():
            entitlement_list.append({'id': entitlement['id'], 'name': entitlement['name']})
    print(str(len(entitlement_list)) + " entitlements found.")
    return entitlement_list
52

Hi,
Thanks for share the script. I was able to use this script. But getting an error, when I run the script. Update role is failing with no error information. Any suggestions?

""Replacing source Name with sourceId in roleAssignValue String
Role ‘Merchandising-Buyers’ update is a Failure
Error:

C:\Work\SailPoint\APIs-RUBY\Role-Importer\Role-Importer>“”

Thanks

53

I have tested v8.0.1 and I can see that is connect to /beta/
But it only return 250 entitlement in total. Was that the intension?
Total number of Entitlements of this source (ID 000000000000000000000): 250

54

Hi Karsten, unfortunately that’s because the script uses a cursor which SP have deprecated. I’ll get around to posting an updated version one of these days if needed but @M_rtenH’s code above should provide you with the logic needed if you wish to update the script yourself.
Or you can replicate the code blocks used for get roles and get APs.
It’s a shame SP have deprecated the cursor as it was quite helpful in keeping code tidy and concise.

55

Thanks Marten for your response. Actually obtaining the Accounts data from the GUI is just slow to obtain especially when you are tryng to do it for 100+ apps one by one. The search doesn’t get the RAW account data we need either and I agree that using Postman and APIs is limited. Unfortunately as I am not a developer by trade and no one else on my team is, we are unable to figure out how to use PowerShell or Python to do the necessary. We literally don’t understand what is required software wise and lastly if we obtained the script, what we would need to change to ensure it works against our environment and export it in the format we need it. API was easy for us before the limits applied but at least it was easy enough to use and could be automated without human triggers.

I just wish someone from the Developer community would share us the Alternatives of using APIs with PowerShell and else and provide us with out of the box ready to go packages which we only need to change a few details like tenant and authentication and then viola, job done we are back to normal and we can get what we need.

But unfortunately I don’t see that happening and so we will just have to waste time exporting it from the front end GUI and we will over time just re-consider other tools over Sailpoint in the future as they are forever changing what I loved about the tool, but less and less everyday.

56

Yeah - if you’re building a report for all applications that’s a different story.
Pagination being enforced on endpoints is fairly standard but it’s a shame there’s no alternative to do it in bulk via UI for example (i.e standard “All source accounts” report that you could run).

I don’t have a script ready to share (at least none that would be as plug and play as you describe) but I’ll try to remember if I ever end up needing one.

1 Like
57

Trying to build one using CHAT GPT now. Just can’t figure out one issue currently which is adding in
the filters because the inverted comas are disrupting the flow is what I believe. Its coming up with an unexpected token error for the Source ID eq part. And not sure how to overcome it

’

Define the base URL of the API and the endpoint to be queried

$baseUrl = “https://tenant.api.identitynow.com/v3/accounts?filters=sourceid eq”
$endpoint = “/items”

Define the URL to retrieve the JWT token from

$jwtUrl = “https://tenant.identitynow.com/ui/session/get-jwt”

Retrieve the JWT token from the URL

$jwtToken = Invoke-RestMethod -Method Get -Uri $jwtUrl -ContentType “application/json”

Define the source ID

$sourceId = “2c9180867a15f0f1017a1f6580392a5f”

Construct the full base URL with the source ID

$fullBaseUrl = “$baseUrl "$sourceId"”

Define the offset value

$offset = 0

Define the maximum number of results to retrieve in each iteration

$limit = 250

Set a flag to determine when to stop the loop

$moreResults = $true

Create an empty array to store the results

$results = @()

Start a loop to retrieve the results in chunks of 250

while ($moreResults) {

Construct the full API URL with the offset and limit parameters

$apiUrl = “$fullBaseUrl$endpoint?offset=$offset&limit=$limit”

Set the Authorization header with the JWT token

$headers = @{
“Authorization” = “Bearer $jwtToken”
}

Make the API request and store the response in a variable

$response = Invoke-RestMethod -Method Get -Uri $apiUrl -Headers $headers -ContentType “application/json”

Check if there are more results to retrieve

if ($response.total -le $offset + $limit) {
$moreResults = $false
}

Add the results to the array

$results += $response.items

Increment the offset value

$offset += $limit
}

Do something with the final array of results

$results | ForEach-Object
’

image
image701×154 5.72 KB

58 
The below line needs to be after, $sourceId = “2c9180867a15f0f1017a1f6580392a5f”
baseUrl = “https://tenant.api.identitynow.com/v3/accounts?filters=sourceid eq $($sourceid)”

Then you don't need - $fullBaseUrl = “$baseUrl "$sourceId"”

Of course you'll need to fix the apiUrl line:
$apiUrl = “$($baseUrl)$($endpoint)?offset=$($offset)&limit=$($limit)”
1 Like
59

Worth mentioning, even if it only helps out 1 other person, but Version: 8.5.1 of the importer-Exporter tool fixes this issue RE: paging limitation for the tool.

1 Like