V3 search API limit defaulting to 250 records

[sup3rmark (Mark Corsillo)]

Hello!
I know from the documentation (and a number of comments on this forum mentioning as much) that the default (and max) limit for calls to the v3 search endpoint is supposed to be 10,000, but when I try to make a call to that endpoint with no specified limit, I only get 250 results; when I try to specify a limit of 10000, I get a 500 response saying A downstream resource was unavailable.

I’m trying to request all identities in one go, so the request I’m sending is:

{
  "query": {
    "query": "*"
  },
  "sort": [
    "id"
  ],
  "indices": [
    "identities"
  ]
}

I get the same error with a limit of 5000.

Note: if I add "includeNested": false to my query, I can successfully query with limit=10000 explicitly defined, but if I don’t include that parameter I still only get 250 records returned.

I know including all the nested results adds a lot to each call, especially if I’m not specifying a queryResultFilter, so a full search for 10k identities with nested objects and all attributes may just be too much for the server to handle, but figured it was worth posting about this here since it does result in a deviation from the documented behavior. Not sure if this would be considered a bug, or if it would just be worth mentioning in the docs…

-Mark

[baoussounda (Ousmane N'DIAYE)]

@sup3rmark

ownstream resource was unavailable.

Hi @sup3rmark,

Agree with you regarding the default limit value for search endpoint. It seems there might be an error since the documentation states it as 10,000, whereas for all other API calls, the default value of the limit parameter is 250.

I have encountered this error before, where the server returned “downstream resource was unavailable,” likely due to the large size of the data being retrieved. This could potentially be considered a bug, as it results in a server error. Even if the server manages to return the data, it’s doubtful whether the client would have sufficient resources to process all of it.

As you are aware, identities have a multitude of data (accounts, access, apps, etc.). One solution to mitigate the issue of excessive data size returned by a search is to use includeNested (you already tested this). Another approach could involve using the includes or excludes options of the “queryResultFilter.”

{
  "query": {
    "query": "*"
  },
  "sort": [
    "id"
  ],
  "indices": [
    "identities"
  ],
  "queryResultFilter": {
    "includes": [
      "name",
      "displayName"
    ]
  }
}

[colin_mckibben]

This definitely sounds like a bug. Search is supposed to return up to 10,000 records per page. If you can reliably reproduce this issue, it is worth opening a support ticket to have them investigate.