Webservice Pagination not working for NERM API

SailPoint Human Fabric (SHF) SHF Discussion and Questions
, , ,
1

Hi All,

I’m working on pagination of NERM API where i used metadata=true to get the next set of accounts.

TERMINATE_IF $RECORDS_COUNT$ < $pageSize$ OR $response._metadata.next$ IS NULL OR $response._metadata.next$ == ""
$endpoint.fullUrl$ = $application.baseUrl$ + $response._metadata.next$

But when I try to aggregate, the count is 200, but the actual record came into SailPoint as 100, but I have 131 accounts total. Any idea how to resolve this?

2

Check your API (response strcuture carefully) , and ensure thatSailpoint is mappings are: record path - accounts ; total count path - metadata.count; next page path -metadata.next; stop condition next == null

3

Did I need to add the metadata in response mapping? Or what exactly you are saying?

4

yes, mate SP must know where is the current account array is, where the next - page info is and where the total account is…BUT ONLY THE ACTUAL ARRAY (ACCOUNTS) SHOULD BE AGGREGATED

5

I have added the profiles.* in the root path of response information. But the metadata.next is coming from outside of these profiles.

image
image389×205 3.13 KB

Can you help me with what exactly I need to do?

6

Try this Response attribute mapping ( $.profiles (Root Path) ; Pagination Attribute $._metadata.next; Total account (Optional) $._metadata.total and the offset key offset

7

so to fix this YOU MUST map the pagiantion variables relative to the absolute root ($) while the account attributes remain relative to the Root Path ($.profiles)

8

Like this below, right? anything need to change in pagination script

image
image190×257 1.97 KB

image

image
image628×187 2.72 KB

TERMINATE_IF $RECORDS_COUNT$ < $pageSize$ OR $response._metadata.next$ IS NULL OR $response._metadata.next$ == ""
$endpoint.fullUrl$ = $application.baseUrl$ + $response._metadata.next$
9

Hello

I don’t know why you are using metadata, but check this post to configure pagination in NERM Push or Pulled IDN users into NERM via API - Identity Security Cloud (ISC) / ISC Discussion and Questions - SailPoint Developer Community

10

Hi @srezzaki, I already tried this, resulting in same issue i faced earlier: aggregated count 200 but actual records 100

11

The configuration should definitly work, can you share the configuration of the aggregate operation and results (screenshots)

12

Sure here it is.

image

image
image338×178 3.48 KB

image
image786×540 9.76 KB

image
image1040×442 12.7 KB

13

Use variables in context url “limit=$response._metadata.limit$&offset=$sysparm_offset$” instead of limit=0&offset=100

14

Changed to context url now the aggregation is is running for more than 20 min

image

15

Hi @Santhakumar ,

Why don’t you use the direct NERM connector instead of the WebService-based?

image
image1059×787 29.2 KB

Anyway, you need to use a webservice; use the following pagination configuration.

Initial Page Offset = 0

Page Size = 50

Paging Steps

TERMINATE_IF $RECORDS_COUNT$ < $limit$
$offset$ = $offset$ + $limit$
$endpoint.fullUrl$ = $application.baseUrl$ + $endpoint.relativeUrl$ + “&limit=” + $limit$ + “&offset=” + $offset$

Let me know how it’s worked.

Thanks,

@SivaLankapalli

16

@SivaLankapalli @srezzaki @jorgedeloittespain Thanks for your inputs. I have tried the pagination script, but it is not working as expected for NERM APIs. At last I found the solution.

What have I tried so far?

  • Set the root path as $ and add the pagination attributes to store metadata values like total, offset, and next.
  • Tried different ways to configure the pagination script as below.
TERMINATE_IF $RECORDS_COUNT$ < $pageSize$ OR $NO_RECORDS$ == TRUE
$offset$ = $offset$ + $pageSize$
$endpoint.fullUrl$ = $application.baseUrl$ + "/api/profiles?metadata=true&limit=" + $pageSize$ + "&offset=" + $offset$ + "&profile_type_id=<profileid>"

TERMINATE_IF $RECORDS_COUNT$ < $pageSize$ OR $response._metadata.next$ IS NULL OR $response._metadata.next$ == ""
$endpoint.fullUrl$ = $application.baseUrl$ + $response._metadata.next$

$sysparm_final$ = $response._metadata.limit$ + $response._metadata.offset$
TERMINATE_IF $sysparm_final$ == $response._metadata.total$
$sysparm_offset$ = $response._metadata.limit$ + $response._metadata.offset$
$endpoint.fullUrl$ = $application.baseUrl$ + "/profiles?metadata=true&profile_type_id=<profileid>&limit=" + $response._metadata.limit$ + "&offset=" + $sysparm_offset$
  • Mapped the variable in the context URL along with the pagination script as the 3rd one above.
/api/profiles?metadata=true&limit=$response._metadata.limit$&offset=$sysparm_offset$&profile_type_id=<profileid>

But none of them works for NERM APIs. The reason behind this is the NERM API with metadata=true as a parameter will give the response as below.

image
image389×205 3.13 KB

Since the metadata.next keep goes on to 100,200, if we use it in a pagination script, it will be considered a loop, and the aggregation will run forever. The pagination script will never work even if you try with different settings.

Solution

  • Remove the /api from baseUrl
  • Add the /api in the context url of the Account Aggregation Operation as below
/api/profiles?&profile_type_id=<profileid>
  • Now add the metadata, limit and offset to the context url as below
/api/profiles?metadata=true&limit=$response._metadata.limit$&offset=$response._metadata.offset$&profile_type_id=<profileid>

/api/profiles?metadata=true&limit=$response._metadata.limit$&offset=$sysparm_offset$&profile_type_id=<profileid>

You can use either one of the above codes; both will work.

Thanks.

17

This is not working for advanced search api for web service connector Saas.

18

Metadata for the Advanced Search API hasn’t worked for years… Expected but not desirable, I suspect changes will eventually come for this.