Web Services SaaS Connector - Paging Help

[SeanK-W (Sean Krieger-Will)]

Hi ISC Community,

We’re trying to setup an Account Aggregation for our Web Services SaaS connector.

We successfully pull 2000 objects; however, it stops there… This is what we’re using at the moment as our paging steps (work in progress):

TERMINATE_IF $pageIndex$ > 11
 
$pageIndex$ = 0
 
$endpoint.fullUrl$ = $application.baseUrl$ + "/Api/v1/People/Search?External=false&ViewModelType=1&IsDeleted=false&SortField=lastname%2C%20firstname&SortOrder=asc&PageSize=1000&PageIndex=" + $pageIndex$
 
$pageIndex$ = $pageIndex$ + 1

We would expect to see around ~10000 accounts:

We’re following the documentation, however struggling to see any guidance on how to set up paging to iterate through the dataset until no further records: Aggregation Paging

Does anyone have any advice as to what paging steps we need to include?

Cheers,

Sean

[vkashat (Vincent Kashat)]

I don’t think you can use $pageIndex$, you’ll have to one of the predefined keywords.

Try using the limit/offset paging, set the limit to 1000, offset to 1, and this:

TERMINATE_IF $RECORDS_COUNT$ < $limit$
$endpoint.fullUrl$ = $application.baseUrl$ + "/Api/v1/People/Search?External=false&ViewModelType=1&IsDeleted=false&SortField=lastname%2C%20firstname&SortOrder=asc&PageSize=$limit$&PageIndex=" + $offset$
$offset$ = $offset$ + 1

[SeanK-W (Sean Krieger-Will)]

Thanks @vkashat ,

Tried this - but sadly it still stops at 1000 accounts

***EDIT: Additionally, if I was to remove all paging steps. It still only pulls 1000?

[shirbhatea (Ankush Shirbhate)]

can you add the request body here, if you are using it?

[zeel_sinojia]

Hey @SeanK-W,

Weird that the case you gave for pagination worked for 2k accounts. I think it should have ran until the aggregation times out as the $pageIndex$ variable is set to 0 everytime it runs.

Having said that, you can give the following pagination logic a try.

TERMINATE_IF ($response.PageSize$ <= 0)
$pageIndex$ = $response.CurrentPage$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + "/Api/v1/People/Search?External=false&ViewModelType=1&IsDeleted=false&SortField=lastname%2C%20firstname&SortOrder=asc&PageSize=1000&PageIndex=" + $pageIndex$

1. The Termination logic is not hard coded, so it can handle even more accounts if added in the future.
2. The pageIndex is auto calculated based on the response from the request. That way we don’t have to worry about the data stored in the variable during the last iteration. Whatever current page is, we want the next page of that as long as there’s data.

HTH

[SeanK-W (Sean Krieger-Will)]

Hey @zeel_sinojia , thank you! It has almost worked!

Good news is, this does seem to read all of the accounts, bad news is - it gets stuck aggregating and doesn’t seem to terminate. Any ideas on where we’re going wrong?

Have also tried these attempts, sadly no success. All read the accounts, but fail to terminate the aggregation:

TERMINATE_IF ($response.Results.size() <= 0)

$pageIndex$ = $response.CurrentPage$ + 1

$endpoint.fullUrl$ = $application.baseUrl$ + "/Api/v1/People/Search?External=false&ViewModelType=1&IsDeleted=false&SortField=lastname%2C%20firstname&SortOrder=asc&PageSize=1000&PageIndex=" + $pageIndex$

&

TERMINATE_IF ($response.Results.size() == NULL)
$pageIndex$ = $response.CurrentPage$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + "/Api/v1/People/Search?External=false&ViewModelType=1&IsDeleted=false&SortField=lastname%2C%20firstname&SortOrder=asc&PageSize=1000&PageIndex=" + $pageIndex$

&

TERMINATE_IF (NO_RECORDS == TRUE)
$pageIndex$ = $response.CurrentPage$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + "/Api/v1/People/Search?External=false&ViewModelType=1&IsDeleted=false&SortField=lastname%2C%20firstname&SortOrder=asc&PageSize=1000&PageIndex=" + $pageIndex$

[SeanK-W (Sean Krieger-Will)]

Nvm, I’m a muppet. I forgot the $ in my last test.

This worked

TERMINATE_IF $NO_RECORDS$ == TRUE

$pageIndex$ = $response.CurrentPage$ + 1

$endpoint.fullUrl$ = $application.baseUrl$ + "/Api/v1/People/Search?External=false&ViewModelType=1&IsDeleted=false&SortField=lastname%2C%20firstname&SortOrder=asc&PageSize=1000&PageIndex=" + $pageIndex$

Thanks everyone!