Web Services Connector | Pagination through "continuationToken in response body"

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

Hi @colin_mckibben , IdentityNow Experts,

There’s one target we’ve configured through Web Services Connector. It supports continuationToken for pagination. This is how it works:

  1. Request body of first call would not have continuationToken and it will return 100 records.
  2. Next request body would have continuationToken received from first response body.
  3. Calls will continue until we stop receiving continuationToken in response body.

It works well through Postman calls but when we’re trying to achieve the same through Paging in IdentityNow Web Services Connector, we notice that:

198 accounts are scanned but only 99 accounts are displayed under “Accounts” tab after “Account Aggregation”. However total number of accounts at target are 300+

Screenshot for “Accounts” tab:

image

Screenshot from “Aggregation Activity Log”:

image
image916×105 6.32 KB

Paging Steps we’ve added:

TERMINATE_IF $response.continuationToken$ == NULL
$endpoint.fullUrl$ = $application.baseUrl$ + “contextUrl”
$request$ = “{ “attr1”: “value1”, “continuationToken”: “$response.continuationToken$”}”

This is how we’re adding continuationToken in request body:

{
“attr1”: “value1”,
“continuationToken”: “$response.continuationToken$”
}

Could you please advise on this?

Your quick response would be highly appreciated.

EDIT:

We’ve just noticed that it’s iterating same 99 accounts twice that’s why we see number of accounts scanned as 198 and actual accounts under “Accounts” tab are 99.

It seems that continuationToken is not being read/used during pagination.

2

Have you tried

$request$ = “{ \“attr1\”: \“value1\”, \“continuationToken\”: \“$response.continuationToken$\”}”
3

Thanks for quick response, @iamnithesh ,

I tried that but issue remained same, it scanned 198 accounts and displayed 99 accounts under Accounts tab while target has 300+ accounts.

Any other suggestions ?

4

Can you remove above line and run an aggregation? Do you get the same result as you are getting now?

If you get any error please post the screenshot of the same

5

Hi @iamnithesh ,

Many thanks for responding back.

We ran account aggregation after removing line “TERMINATE_IF $response.continuationToken$ == NULL” from paging steps and results remained same as it scanned 198 accounts and displayed 99 accounts under Accounts tab while target has 300+ accounts.

Account aggregation goes successful, there’s no error in “Aggregation Activity Log”.

We checked ccg.log file and noticed that there’s no error/exception there as well but connector iterator is being closed before execution of all tasks in queue. Sharing message details from ccg.log file:

.
.
.
"class":"com.sailpoint.aggregation.server.service.extract.BaseExtractor","clientId":"****","source_host
":"****","method":"logFinishExtraction","org":"****","level":"INFO","IdentityIQ":"8.0 Build ****,"message":"Closing connector iterator for source: ****, iteration Exception: false, count: 198, duration: 395 ms",
"pipeline":"****","@timestamp":"2023-07-06T02:13:14.739Z","thread_name":"****","metrics":"****","region":"****","AppTyp
e":"Web Services","Application":"****".........
.
.
.
........."class":"com.sailpoint.aggregation.server.util.TaskBatcher","clientId":"****","source_host":"****","method":"c
lose","org":"****","level":"INFO","IdentityIQ":"8.0 Build ****","message":"**Found 148 tasks still in qu**
**eue for account-processor-aggregate-batcher**","pipeline":"****","@timestamp":"2023-07-06T02:13:14.930Z","thread_name":"****",
"metrics":"****","region":"****","AppType":"Web Services","Application":"****","request_id":"****".........
.
.
.
........."class":"com.sailpoint.ccg.handler.StreamingAggregationHandler","clientId":"****","source_host":"****","method":"logAggregationStats","org":"****","level":"INFO","IdentityIQ":"8.0 Build ****",
"message":"{\"sourceId\":\"****\",\"sourceName\":\"****\",\"totalAccounts\":198,\"totalTim
eMillis\":1620,\"timeMillisPerAccount\":8.18,\"error\":null,\"aggregationResult\":{\"startDate\":\"Jul 6, 2023, 2:13:14 AM\",\"endDate\"
:\"Jul 6, 2023, 2:13:14 AM\",\"maxMessagesExceeded\":false,\"stats\":{\"total\":198,\"optimized\":0,\"deleted\":0,\"changed\":0,\"added\
":198}.........

Any thoughts/suggestions on this?

6

HI @iamnithesh , Team,

We’ve just noticed that it’s iterating same 99 accounts twice that’s why we see number of accounts scanned as 198 and actual accounts under “Accounts” tab are 99.

It seems that continuationToken is not being read/used during pagination.

Could you please advise on this?

7

Hi @ikram_momin ,

Could you please advise on this case ?

8

Try this.

TERMINATE_IF $NO_RECORDS$ == TRUE

9

Thanks for quick response, @ikram_momin .

I tried that but issue remained same.

Any other suggestions ?

10

Can you please share sample response to see format of continuationToken ??

11

Here you go:

{
    "users": [
        {
           "user1Attr1": "value1"
        },
		{
           "user2Attr2": "value2"
        },
		{
           "user3Attr3": "value3"
        }
		
		],
		"continuationToken": "******"
}
12

Instead of
$request$ = “{ “attr1”: “value1”, “continuationToken”: “$response.continuationToken$”}”

try
$request.continuationToken$ = $response.continuationToken$

5 Likes
13

Thanks a million, @ikram_momin .

It worked!

1 Like
14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.