Issues with WebServices SaaS connector

[aditya_pathak]

Hi community,

I have a scenario wherein i am encountering multiple issue with the WebServices SaaS connector.
a) I have a custom authentication setup as the first call which stores access_token at an application level
b) second call is for test connection wherein i pass header with -
Authorization → $application.accesstoken$ - works
c) I am using a similar call for aggregation (as test connection) and storing the response in schema attributes - facing 1st issue here - aggregation result shows success with 69971 accounts created , however when i check the accounts page it just lists only 10k records -

on clicking on accounts it changes to 10k

it doesn’t even change with adding pagination logic, aggregation always says accounts created but nothing really shows up.

d) I have to run another call in chain using an attribute in previous call and im providing the context url in the usual format -

with parent endpoint configured but get an error
Your client has issued a malformed or illegal request.

have checked the format is correct via postman so im thinking the value is not getting replaced . i have also checked the csv extract and the personNumber is populated there correctly.

Any tips.

[ipobeidi (Ivan Obeidi)]

Hey @aditya_pathak , how are you?

When this change happen " it gets 69k objects but it changes to 10k" means that you’re only getting 10k unique accounts.

That could mean that ISC is repeating the same query, pagination errors, account attribute mapping errors.

did you configure any of it?

[aditya_pathak]

hi @ipobeidi -i have tried with or without pagination , also tried a hardcoded body(increasing index) next in aggregation sequence (without a parent) which gives me next page in postman , but here it seems to only get those 10k records

[ipobeidi (Ivan Obeidi)]

Can you share the Pagination you’ve done?

[sivakrishna_1993 (Sivakrishna Kommineni)]

Hello @aditya_pathak,

In the configuration seting enable Use hasMore Attribute for Aggregation Termination.
Then run the aggregation.

Thanks,
Siva.K

[aditya_pathak]

Hi @ipobeidi just to give a bit more context here, a single call returns max of 10k records - its what we see in records in ISC , also total number of records in the source are around 29k → so not sure how the connector gets the 69k records from and says 69k accounts created (refer earlier screenshot) . I was thinking more on the same lines that my pagination logic is not working and have tried a few pagination variants but no luck.

How this api next page works -
basically the context url remains same and next page is traversed through an index in body such as
{
“index”: 0
}

the response returns a “totalElements” attribute with a max value of 10k and returns “totalElements”: “0” when no elements are found. Hence the reason as a test, I tried the next aggregation call in sequence without chaining with hard coded values(index 1) but the connector doesn’t seem to pull in the next set of accounts.

Although I have tried various variants of pagination logic , the last version used is -
TERMINATE_IF $response.totalElements$ == 0
$index$ = $request.index$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + “/api/v1/commons/persons/apply_read”
$request.index$ = $index$

Strangely, on reset and reaggregate it now seems to pull in more (this is through pagination logic as I had removed that hardcoded sequential call)-

maybe a bug ?

[aditya_pathak]

thanks Siva, though my API doesnt use hasMore. refer above comment

[ipobeidi (Ivan Obeidi)]

Thanks , one more question. Does the URL accept variables? I saw you didnt use in the pagination any variable to go for the next page. like this :

$page$ = 1
TERMINATE_IF $response.hasNextPage$ == false
$slimit$ = 100
$page$ = $page$ + 1
$endpoint.fullUrl$ = $application.baseUrl$ + " user/paginated?limit=100&page=" + $page$

best

[aditya_pathak]

@ipobeidi as mentioned earlier,
How this api next page works -
basically the context url remains same and next page is traversed through an index in body such as
{
“index”: 0
}
, so next pages will have index 1,2 and so on passed through the body and not url

[aditya_pathak]

@philip-ellis any suggestions here ?

[ipobeidi (Ivan Obeidi)]

In that case, you’re better with a After Operation Rule that can call the APi changing the request body.

that would be the best option for your use case.

[MVKR7T (Krishna Mummadi)]

I have seen similar issue recently, I am not sure if you have multiple HTTP operations for Aggregation, In the 2nd Aggregation operation maybe it is getting 10K accounts only. Remove that and see if you are getting all the records, post that fix the 2nd operation.

[aditya_pathak]

hi @MVKR7T - If you read my original post i do have 2 aggregation operations chained. with the 2nd operation linked the aggregation immediately fails. I can get all the accounts through the first aggregation and can check in the CLI logs that the $response.personNumber$ value is substituted for the first record (in the chained call) but still failing with Your client has issued a malformed or illegal request., so can only think that maybe the Authorization which is working on the first call is somehow not working in the second call. Have you seen similar issues with Custom Auth ?

[MVKR7T (Krishna Mummadi)]

No, if custom auth is working for one call, it work for others too if you added the header correctly.

@aditya_pathak

Your client has issued a malformed or illegal request.

This doesn’t mean that authentication is not working. You need to check the configurations of both the calls.

Can you confirm without 2nd call added, you can aggregate all the accounts ?

[sauvee (Edwin Sauve)]

@ipobeidi

In that case, you’re better with a After Operation Rule that can call the APi changing the request body.

that would be the best option for your use case.

Note that you cannot use Before or After Operation Rules with the SaaS based Web Services Connector, only with the VA based one!

Any logic fit for a before or after operation rules need to be added into a customizer:
https://documentation.sailpoint.com/connectors/saas/web_services/help/SaaS_Connectivity/Web_Services/Customizers.html

[aditya_pathak]

yes agg call 1 works without issue and If i hardcode the second call with bearer token and personNumber values it works for one user. As already mentioned I can see $response.personNumber$ value is substituted in the normal scenario before it fails with the error. The auth header is same for call 2 as call 1 Authorization → Bearer $application.accesstoken$

[aditya_pathak]

thanks Edwin - though i was able to achieve it through pagination script