Web Service Connector|Pagination Through queryToken in Response Body

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

Hi IdentityNow Experts ,

here’s one target we’ve configured through Web Services Connector. It supports queryToken for pagination. This is how it works:

  1. Request body of first call would not have queryToken and it will return 100 records.
  2. Next request body would have queryToken received from first response body.
  3. Calls will continue until we stop receiving queryToken in response body.
  4. Root Path to get the accounts is QueryResult/results but query token is under QueryResult so not able to get queryToken in response as the Root path is QueryResult/results. Sample respone is as below - {
    “@type”: “QueryResult”,
    “queryToken”: “*****”,
    “result”: [
    {
    “@type”: “AccountUserRole”,
    “id”: “id1”,
    “accountId”: “abc”,
    “userId”: “uerid1”,
    “roleId”: “roleid1”,
    “lastName”: “”,
    “firstName”: “”
    },
    {
    “@type”: “AccountUserRole”,
    “id”: “id1”,
    “accountId”: “abc”,
    “userId”: “userid1”,
    “roleId”: “roleid2”,
    “lastName”: “”,
    “firstName”: “”
    },
    {
    “@type”: “AccountUserRole”,
    “id”: “id2”,
    “accountId”: “abc”,
    “userId”: “userid2”,
    “roleId”: “roleid3”,
    “lastName”: “”,
    “firstName”: “”
    },
    {
    “@type”: “AccountUserRole”,
    “id”: “id2”,
    “accountId”: “abc”,
    “userId”: “userid2”,
    “roleId”: “roleid1”,
    “lastName”: “”,
    “firstName”: “”
    }
    “numberOfResults”: 435
    }

Can you please suggest how the pagination will work in this case?

2

Are you sure of this? Usually such tokens are returned in the first response itself. I cannot see how you can make a second call without queryToken

3

Hi @sawhnr01

The paging engine can normally access to the entire response body through $response, even when the Root Path points to the $.results in your case

Could you confirm where the queryToken needs to be sent for the next request? does it need to be included in the request body or request url as query param ?

You could try configuring the paging steps like this if the recevied queryToken should be added into the next request Body :

TERMINATE_IF $response.queryToken$ == NULL

$request.queryToken$ = $response.queryToken$

This will stop paging when no queryToken is returned and populate the queryToken in the next request body with the value received from the previous response.

If your next request body has a different structure, update the paging logic accordingly. $response.queryToken$ should always retrieve the token from the previous response.

4

Yes, you are right . The querytoken i get in first reponse as well. but the response from the API looks like this:

xxxxxxxx

<result …/>
<result …/>
<result …/>
…

The challenge is with the Response Information Root Path.

Option 1: Root Path = /QueryResult/result

the connector behaves as expected and creates one resource per element, for example:

{
Resource 1

userId = user1
roleId = role1

Resource 2

userId = user1
roleId = role2

Resource 3

userId = user2
roleId = role3
}

This allows account aggregation to work correctly. However, since the root path starts at , the queryToken is no longer available to the connector or the After Operation Rule, so I cannot use it for pagination.

Option 2: Root Path = /QueryResult

I can access the queryToken, which is required for calling the next page.

However, the connector now treats the entire as a single resource. The mapped attributes become arrays, for example:

{

userId = [a, a, a, a, b, b, b, c]

roleId = [1, 2, 3, 4, 7, 8, 9, 5]

}

Instead of individual resources, all values are returned as lists within a single resource.

This means I would have to manually correlate the userId and roleId arrays in an After Operation Rule and reconstruct the individual account resources before aggregation can continue.

Has anyone implemented pagination for a similar API where:

the pagination token (queryToken) exists outside the repeating elements,
but the account records are inside ?

Is there a recommended approach in the Web Services connector to:

Aggregate accounts using /QueryResult/result, and
Still access the parent-level queryToken for pagination,

without having to rebuild all resources manually in an After Operation Rule?

5

I tried this approach but doesn’t work as the API is different after the first run . I tried updating endpoint URL as well using pagination but no luck .

6

What is the API format after the first run ?

7

first API is like Baseurl/AccountUserRole/query and 2nd becomes baseurl/AccountUserRole/queryMore and for 2nd url which is baseurl/AccountUserRole/queryMore the body should be the queryToken from the 1st API response. And the 3rd API will be the same as2nd but the Body will be the queryToken from the 2nd API response.

8

@sawhnr01 have you tried this :

TERMINATE_IF $response.queryToken$ == NULL

$endpoint.fullUrl$ = $application.baseUrl$ + "/AccountUserRole/queryMore"

$request.queryToken$ = $response.queryToken$
9

yep already tried this .

10

We have similar endpoints and it work for us.

The root path is configured like this $.data.employees:

image
image844×408 24.5 KB

And the pagination logic :

TERMINATE_IF $response.nextpagetoken$ == NULL
$next$ = $response.nextpagetoken$
$endpoint.fullUrl$ = $application.baseUrl$ + "/services/api/x/users/v2/employees?nextpagetoken=" + $next$

Our sample json look like :

{
   "nextpagetoken":"exxeexxx..",
   "data" : {
"employees": [ {},
     {}]

     },
"total":5000
}

The pagination works fine and the only difference in your case is that should set the token in the next request body

$request.queryToken$ = $response.queryToken$

The following post confirmed that the token can be set to the next request body Web Services Connector | Pagination through “continuationToken in response body” - Identity Security Cloud (ISC) / ISC Discussion and Questions - SailPoint Developer Community

11

Might be the issue is that the response i am getting is in XML not JSON.

12

Can you please describe more your XML response format ? And XML request body format ?

13

Does your vendor API have any capability to provide JSON?
i.e. by sending the header (“Accept”, “application/json”)

If not, you may need to look into a WebServicesAfterOperation to parse the XML and set a hasMore and a transientValue of that item
and a WebServicesBeforeOperation rule to check the transients and utilize that data.

14

youre mileage may vary, but heres a couple pieces:

AFTER OPERATION RULE

Map transientValues = application.getAttributeValue("transientValues");
if(transientValues == null) {
  transientValues = new HashMap();
  application.setAttribute("transientValues", transientValues);
}
[...]
//find the data, and store it
       transientValues.put("ArbitraryTokenName", TheToken);
transientValues.put("hasMore", linkFound); //note: has to be hasMore, and true/false`

BEFORE RULE:

Map tMap = (Map) application.getAttributeValue("transientValues");
if(null != tMap) {
  String theToken = tMap.get("ArbitraryTokenName");
  log.debug("nextLink : " + theToken);
  if(!Util.isEmpty(theToken)) {
//use it. example here sets a URL, but... your mileage may vary
    requestEndPoint.setFullUrl(theToken);
  }
}
return requestEndPoint;