I currently have a simple python script that authenticates and then makes a call to the entitlements API. It works fine iterating through the offsets with a limit of 250. Once the code reaches offset 10000 it fails with a 400 error.
I’ve ran queries through the CLI to get the data, so I know it works.
The code starts at offset=0&limit=250 and continues to work fine until offset reaches 10000.
Then it errors with the following message.
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url:
?offset=10000&limit=250
The CLI works fine with the offset of 10000:
In searching the forum, I can’t find anything to describe any limit I might be reaching.
a rate limiting error should be 429, so it’s not that.
The duration of the token isn’t a problem, if I short circuit the code to just run with the offset of 10000 from the beginning, it has the same failure.
What you’re hitting is the 10,000 record ceiling on offset-based pagination. Once offset + limit goes past 10,000, the API returns a 400 rather than a 429, which is why it doesn’t look like rate limiting and why it fails even when you start directly at offset 10000. The Search API docs mention it explicitly: to go beyond 10,000 records you need searchAfter paging. The same behaviour has been reported on this thread.
A couple of options:
Switch to the Search API with searchAfter on the entitlements index. Sort on a unique field (id), and pass the last id of each page as searchAfter in the next request:
body = {
"indices": ["entitlements"],
"query": {"query": "*"},
"sort": ["id"]
}
results = []
while True:
r = requests.post(f"{base}/v3/search?limit=250", headers=headers, json=body)
r.raise_for_status()
page = r.json()
if not page:
break
results.extend(page)
body["searchAfter"] = [page[-1]["id"]]
Keep the entitlements endpoint but partition the calls with a filter, for example filters=source.id eq "...", so each slice stays under 10,000. Handy if you need the exact shape returned by /entitlements rather than the search document.
As for the CLI working, it most likely isn’t paging the same way your script does. Running it with --debug will show you the actual requests it sends.
Hi guys,
Thank you for the replys. I’ll refactor the code to deal with these limitations. Seems like some missing detail in the documents. I hit a similar situation with an ID that has more than 500 entitlements when querying all entitlements by identities.
Happy coding.
Ty
