Bulk download or stream of a Sources Accounts

[aceiss (Bob Hanson)]

Is there an API or programatic way to download a sources accounts? The 250/request limit has issues when trying to get 1000’s or even 10,000s accounts on a source. Some Active Directories can be rather large.

[iamnithesh (Nithesh Rao)]

You can use any scripting language (Ruby, NodeJS, php, etc) that supports connection to REST API and write code to send multiple requests in sequence (using offset: 0, 250, 500, … and limit 250) and save the response one page at a time. However, make sure you use the sorter query parameter as the records tend to repeat in subsequent pages

[aceiss (Bob Hanson)]

@iamnithesh

ds tend to repeat in subsequent

Im aware of the API and the 250 account limit/request.

If I have 10,000 accounts or 100,000 accounts, 250/request isnt useful. The caller would most likely timeout. I was just wondering if there was a way to get this list in a batch request or maybe via CSV stream etc…

[ethompson (Edward Thompson)]

You can try downloading the accounts from the UI (Source->Accounts->CSV). This calls

https://<tenant>.api.identitynow.com/cc/api/source/runAccountsExportReport

Sometimes this can fail or the number of accounts can create a large CSV that is hard to work with (Ex. 35K AD accounts can be about 1GB)

/v3/accounts should work as well. I have used it for at least 30K accounts and have not seen timeouts. It also allows you to do an processing for each set of 250 so you do not need to store everything in memory at once. Here is an example using PowerShell:

function Get-Accounts() {
    param (
        [String]$org,
        [Int]$limit = 250,
        [Int]$offset = 0,
        [String]$filters = "",
        [bool]$list = $false,
        [String]$key = "name",
        [String]$token
    )

    # Get Total
    $uri = "https://$($org).api.identitynow.com/v3/accounts?limit=1&count=true&filters=$filters"
    $result = Invoke-WebRequest -Method GET -Uri $uri -Headers @{Authorization = "Bearer $token" }
    $total = [int]($result.headers."X-Total-Count")[0]
    if ($total -eq 0) {
        return $null
    }

    $mapping = [ordered]@{}
    $accountsList = $()
    do {
        Write-Progress -Activity "Getting Accounts" -PercentComplete ($mapping.count / $total * 100) -Status "$($mapping.count) / $total"
        $uri = "https://$($org).api.identitynow.com/v3/accounts?offset=$offset&filters=$filters"
        $response = Invoke-RestMethod -Method GET -Uri $uri -Headers @{Authorization = "Bearer $($token)" }
        foreach ($item in $response) {
            $mapping[$item.$key] = $item
        }
        $accountsList += $response
        $offset += $limit
    } until ($response.count -lt $limit)

    Write-Progress -Activity "Getting Accounts" -Completed
    if ($list) {
        return $accountsList
    }
    return $mapping
}

If you set -list $true it will return the accounts in an array. By default it will return a hashtable with the name being the key.