Help with WebService Paging

[ralfonse (Rudolf Alfonse)]

Which IIQ version are you inquiring about?

Version 8.4

Share all details related to your problem, including any error messages you may have received.

Hello Fellow IIQ Devs

I am having a little trouble figuring out the right syntax for paging during aggregation a webService aggregation. I am trying to apply the info here Paging Tab Configuration

I am kind of guessing it needs to look something like this? But don’t know how to increment that pageIndex in the aggregation

TERMINATE_IF $response.itemsPerPage$ == 0
$limit$ = 100

$endpoint.fullUrl$ = $application.baseUrl$ + $endpoint.relativeUrl$ +“?PageSize=$limit$” + “&” + “PageIndex=” ??

In Python it looks like this, I need to use the PageIndex and PageSize for the api I am hitting.:

params = {
‘PageIndex’: 1,
‘PageSize’: 200
}

records =

while True:
print(f"Fetching page {params[‘PageIndex’]}…")
response = requests.get(config[‘url’] + ‘Employee’, headers=headers, params=params)

data = response.json()

current_records = data.get('records', [])
records.extend(current_records)

if not current_records:
    print("No more records")
    break
 params['PageIndex'] += 1

print(len(records))

[kjakubiak (Kamil Jakubiak)]

You can set offset to 1 and than in your code just $offset = $offset+1 right afer TERMINATE_IF line. With that you can replace ??? in you URL with $offset and should be good to go.

Something like here

Paging Based on Request Body Configuration

[ipobeidi (Ivan Obeidi)]

hey try to do something like this:

Account aggregation failed

$page$ = 1 TERMINATE_IF $response.hasNextPage$ == false $slimit$ = 100 $page$ = $page$ + 1 $endpoint.fullUrl$ = $application.baseUrl$ + " user/paginated?limit=100&page=" + $page$

let me know the results
Best!

[ralfonse (Rudolf Alfonse)]

I am not getting back the amount of accounts I thought I would. I get around 1400 back with python and only 237 back when doing in sailpoint.

I have this set up for the paging. Is there something off, or logging I can add in there somewhere?

$page$ = 1
TERMINATE_IF $response.hasNextPage$ == false ← I’ve tried with and without this
$limit$ = 100
$page$ = $page$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + $endpoint.relativeUrl$ + “?PageSize=” + $limit$ + “&PageIndex=” + $page$

I tried this too, but also only got 237

$page$ = 0
$limit$ = 100
TERMINATE_IF $response.itemsPerPage$ == 0
$page$ = $page$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + $endpoint.relativeUrl$ + “?limit=” + $limit$ + “&PageIndex=” + $page$

[kjakubiak (Kamil Jakubiak)]

This line is causing problem
$page$ = 0
Because it is being executed with every iteration means you always set your page to 0 and only then incrementing.

I would suggest to use offset variable and initialize it in gui and only later on increment.

[ralfonse (Rudolf Alfonse)]

Thank you

I tried doing that and still did not get the full result. I tried setting Initial Page Offset to 0 and then tried setting it to 1 in the gui.

$limit$ = 100
TERMINATE_IF $response.itemsPerPage$ == 0
$offset$ = $offset$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + $endpoint.relativeUrl$ + “?limit=” + $limit$ + “&PageIndex=” + $offset$

[ipobeidi (Ivan Obeidi)]

Rudolf, my suggestion would be to user the application base URL and “construct” the relative ex:
$application.baseUrl$ + “/rest/user?limit=100&pageIdex=” $pageOffset$

Best!

[ralfonse (Rudolf Alfonse)]

@ipobeidi

$application.baseUrl$ + “/rest/user?limit=100&pageIdex=” $pageOffset$

Thank you, but no luck with that one either. I tried like this

TERMINATE_IF $response.itemsPerPage$ == 0
$offset$ = $offset$ + 1
$application.baseUrl$ + “/api/Employee” + “/?limit=100” + “&PageIndex=” + $offset$

[ipobeidi (Ivan Obeidi)]

try this :

$page$ = 1
$pg_offset$ = 0
TERMINATE_IF $response.itemsPerPage$ == 0
$pg_offset$ = $pg_offset$ + $page$
$application.baseUrl$ + “/api/Employee?limit=100” + “&PageIndex=” + $pg_offset$

I find that using the variable name that Sailpoint uses tend to give some unknow errors.

[ralfonse (Rudolf Alfonse)]

@ipobeidi

$page$ = 1
$pg_offset$ = 0
TERMINATE_IF $response.itemsPerPage$ == 0
$pg_offset$ = $pg_offset$ + $page$
$application.baseUrl$ + “/api/Employee?limit=100” + “&PageIndex=” + $pg_offset$

This thing hates me.

No luck again. I removed the initial page offset and tried this one, but it did not bring back all of them.

$page$ = 1
$pg_offset$ = 0
TERMINATE_IF $response.itemsPerPage$ == 0
$pg_offset$ = $pg_offset$ + $page$
$application.baseUrl$ + “/api/Employee?limit=100” + “&PageIndex=” + $pg_offset$

[ipobeidi (Ivan Obeidi)]

Maybe the Condition is the wrong thing. , try it TERMINATE_IF $NO_RECORDS$

[ralfonse (Rudolf Alfonse)]

@ipobeidi

TERMINATE_IF $NO_RECORDS$

It looks like it is looping two times and then stopping with the following:

$page$ = 1
$pg_offset$ = 0
TERMINATE_IF $NO_RECORDS$
$pg_offset$ = $pg_offset$ + $page$
$application.baseUrl$ + “/api/Employee?limit=100” + “&PageIndex=” + $pg_offset$

I also tried with this and got the same result

$page$ = 1
$pg_offset$ = 0
$pg_offset$ = $pg_offset$ + $page$
$application.baseUrl$ + “/api/Employee?limit=100” + “&PageIndex=” + $pg_offset$

[ipobeidi (Ivan Obeidi)]

if its looping we’re getting somewhere hahaha.
Now we just need to figure the Offset change.

try this 
$page$ = 1
$pg_limit$ = 100
TERMINATE_IF $RECORDS_COUNT$ < $pg_limit$
$pg_offset$ = $pg_offset$ + $page$
$application.baseUrl$ + “/api/Employee?limit=100” + “&PageIndex=” + $pg_offset$

let me know the results

[ralfonse (Rudolf Alfonse)]

@ipobeidi

$page$ = 1
$pg_limit$ = 100
TERMINATE_IF $RECORDS_COUNT$ < $pg_limit$
$pg_offset$ = $pg_offset$ + $page$
$application.baseUrl$ + “/api/Employee?limit=100” + “&PageIndex=” + $pg_offset$

No change on that one. it loops twice and stops. Guessing it is twice because I get 178 back with a limit of 100.

[kjakubiak (Kamil Jakubiak)]

Can you paste whole operation definition part xml?

[ralfonse (Rudolf Alfonse)]

Sure,

                <entry key="bodyFormData"/>
                <entry key="bodyFormat" value="raw"/>
                <entry key="jsonBody"/>
              </Map>
            </value>
          </entry>
          <entry key="contextUrl" value="/api/Employee"/>
          <entry key="curlCommand"/>
          <entry key="curlEnabled">
            <value>
              <Boolean></Boolean>
            </value>
          </entry>
          <entry key="customAuthUrl"/>
          <entry key="header"/>
          <entry key="httpMethodType" value="GET"/>
          <entry key="operationType" value="Account Aggregation"/>
          <entry key="paginationSteps" value="$page$ = 1&#xD;&#xA;$pg_limit$ = 100&#xD;&#xA;TERMINATE_IF $RECORDS_COUNT$ &lt; $pg_limit$&#xD;&#xA;$pg_offset$ = $pg_offset$ + $page$&#xD;&#xA;$application.baseUrl$ + “/api/Employee?limit=100” + “&amp;PageIndex=” + $pg_offset$"/>
          <entry key="pagingInitialOffset">
            <value>
              <Integer>0</Integer>
            </value>
          </entry>
          <entry key="pagingSize">
            <value>
              <Integer>0</Integer>
            </value>
          </entry>
          <entry key="parentEndpointName"/>
          <entry key="resMappingObj">
            <value>
              <Map>
                <entry key="dateOfBirth" value="dateOfBirth"/>
                <entry key="dateOfTermination" value="dateOfTermination"/>
                <entry key="dateofHire" value="dateofHire"/>
                <entry key="department" value="department"/>
                <entry key="division" value="division"/>
                <entry key="email" value="email"/>
                <entry key="employeeNumber" value="employeeNumber"/>
                <entry key="employeeType" value="employeeType"/>
                <entry key="firstName" value="firstName"/>
                <entry key="fullName" value="fullName"/>
                <entry key="gender" value="gender"/>
                <entry key="geographicLocation" value="geographicLocation"/>
                <entry key="homeCity" value="homeCity"/>
                <entry key="homeCountry" value="homeCountry"/>
                <entry key="homePhone" value="homePhone"/>
                <entry key="homeState" value="homeState"/>
                <entry key="homeStreet1" value="homeStreet1"/>
                <entry key="homeZip" value="homeZip"/>
                <entry key="id" value="id"/>
                <entry key="jobPosition" value="jobPosition"/>
                <entry key="jobPositionGroup" value="jobPositionGroup"/>
                <entry key="lastName" value="lastName"/>
                <entry key="loginId" value="loginId"/>
                <entry key="middleName" value="middleName"/>
                <entry key="nickName" value="nickName"/>
                <entry key="supervisor" value="supervisor"/>
                <entry key="workPhone" value="workPhone"/>
                <entry key="workStatus" value="workStatus"/>
              </Map>
            </value>
          </entry>
          <entry key="responseCode"/>
          <entry key="rootPath" value="$.records"/>
          <entry key="sequenceNumberForEndpoint" value="2"/>
          <entry key="uniqueNameForEndPoint" value="Aggregate My App"/>
          <entry key="xpathNamespaces"/>

[ralfonse (Rudolf Alfonse)]

With logging turned on I can see that the right amount of accounts are coming back.

xxx rawResponseObject {
“totalCount”: 1415,

I guess the accounts scanned in the image isn’t the accounts scanned from the end point?

[kjakubiak (Kamil Jakubiak)]

If this is not a secret - could you tell us what is this application? Maybe easiest way would be just if we play around with the paging (or documentation) and we would maybe be able to help you more.

[ralfonse (Rudolf Alfonse)]

Thank you,

It is not. The application is Cority. It is pretty locked down though.

Cority - Award-Winning EHS Software and OHS Software

[ipobeidi (Ivan Obeidi)]

By any chance, is the mapping pulling any non-unique Account ID?