Hi @justinrhaines, because our Search engine leverages Elasticsearch functionality on the backend, you can actually use a searchAfter syntax to get to records beyond the first 10k entries.
The searchAfter capability provides the ability to page on sorted field values instead of offset paging. For instance, if you sort on ID and you page 100 records at a time, after the 1st page of 100 records you can pass the last ID value from the previous record set in your search and then that next search will return 100 records from that previous record matching the ID. You continue that pattern of using the last value passed into searchAfter until the end of the result set. This allows you to page past the 10K limit until the final record is reached.
Here is more information and an example of a Search API call with searchAfter paging:
URL Query Parameters:
limit – passed as URL query param, how many records to fetch per call (250 is the default)
offset – record number to start at in paging, this cannot be used if we utilize searchAfter paging
count - boolean value indicates if you want the total count included in the result header
Search JSON Body (main body):
| indices |
List |
list of indices to which this search should be applied |
| queryType |
String |
should be SAILPOINT which is the format of the query string used in IdentityNow Search UI search box. DSL is also supported but is for advanced uses. (SAILPOINT is default) |
| query |
Query JSON |
this is the Query JSON object, see below for details. (required for SAILPOINT type) |
| sort |
List |
array list of the fields to sort by, this is required if using searchAfter approach, you can use -fieldName for descending searches (optional) |
| searchAfter |
List |
you can use this instead of offset to get past the 10K paging result limit, passing the last value(s) of your sort fields from the previous result set until you retrieve total # of results or end of results. (optional) |
| queryResultFilter |
JSON map |
This optional field allows the user to filter the query result objects by specifying a list of fields to include or exclude from the results JSON. (optional) includes - list of field names including wildcards that should be included to the document result model. excludes - list of field names including wildcards that should be excluded to the document result model. Example:{ “includes”: [ “obj1.", "obj2.” ], “excludes”: [ “*.description” ]} |
Query JSON Object:
| query |
String |
This is where you should pass the query string used in the IdentityNow Search UI search box, you can see Compass documentation for more details on syntax (required) |
| fields |
List |
List of field names to restrict the search provided in query. (optional) |
| timeZone |
String |
Time Zone to be applied to any range query related to dates. (optional) |
Examples using searchAfter:
POST https:// orgname-api.identitynow.com /v3/search?limit=100&count=true { “indices”:[“identities”], “queryType”:“SAILPOINT”, “query”:{ “query”:"*" }, “sort”:[“id”], “searchAfter”:[“2c9180835d38ca0c015d606b50851b1e”] }
POST https:// orgname-api.identitynow.com /v3/search?limit=100&count=true { “indices”:[“identities”], “queryType”:“SAILPOINT”, “query”:{ “query”:“name:Joe AND source.name:“flat file”” }, “sort”:[“id”], “searchAfter”:[“2c9180835d38ca0c015d606b50851b1e”] }
Results will be an array of JSON of the objects matching the search. Also X-Total-Count will be returned in the header with the total # of results because the count=true parameter was passed.