Hey Everyone!
We are working on a Web Services connector and I am having an issue with, what seems to be, pagination termination for account aggregation.
The WS connector calls the initial URL and receives 5000 records along with a new Paging URL in the body that needs to be called separately for subsequent records.
The initial fullUrl looks like this:
https://{baseURL}/Reports/{reportName}
Note: baseURL and reportName are not a true variables, they were just removed for the purposes of this post.
The response body returns data like this:
{
"Data": {
"XRefCode": "{reportName}",
"Rows": [
{
"FirstName": "John",
"LastName": "Smith",
"EmployeeNumber": "0001",
"PersonalEmail": "[email protected]"
},
{
"FirstName": "Jane",
"LastName": "Doe",
"EmployeeNumber": "0002",
"PersonalEmail": "[email protected]"
}
]
},
"Paging": {
"Next": "https://{baseURL}/Reports/{reportName}?cursor=mn%252BRod94XqEEsF0rMrCTCrRzsdBTxAUt4mdlQMXxuC2NSmrjqRPGb0THtW9nGRon"
}
}
The Paging Steps are:
TERMINATE_IF ($response.Paging.Next$ == NULL) || ($endpoint.fullUrl$ == NULL) || ($NO_RECORDS$ == TRUE)
$endpoint.fullUrl$ = $response.Paging.Next$
I started with just the first TERMINATE_IF condition and added the subsequent ones to attempt to fix this issue, but no luck.
The connector is able to parse the first 5,000 records, then the subsequent set of records on the second call (there are currently less than 10,000 records so only 2 calls needed at this time).
However, whenever an aggregation is complete, it shows it as an error with this message:
sailpoint.connector.ConnectorException: Url: , Message: 0 : org.apache.http.client.ClientProtocolException, HTTP Error Code: 0
Looking at the ccg logs, it looks like this is being caused because it is attempting to make another (third) call and the $url is empty, so this error is returned.
==> Dumping request for troubleshooting purposes: HttpRequestWrapper [url=, headers={Authorization=Basic ***, Accept=application/json, Content-Type=application/json}, payload=null, type=GET, allowedSuccessCodes=[0, 2], possibleHttpErrorCodes=, possibleHttpErrorMessages=]"
“Sending request to URL using headers [Authorization, Accept, Content-Type].”
NOTE: Double space in above message after “URL” because $url is empty
“Error while request handling: null, cause is: org.apache.http.ProtocolException: Target host is not specified”
When a call is made that does not have any Paging information, the body returns this for the Paging.Next value:
"Paging": {
"Next": ""
}
So the final successful call that has data returns an empty value for Paging.Next. I know this is different from a true NULL value, but the paging documentation states that a conditional check for NULL indicates it will check for a null or empty object so I would expect paging to terminate upon seeing this empty value returned.
Any thoughts on how to successfully terminate paging in this case to prevent IDN attempting to call an empty url which causes the error message?
Thank you!
- Zach