Web services paging problem

josephcasale · November 11, 2024, 7:32pm

I previously had a web services connector that queried Microsoft Graph and worked just fine. We no longer needed the connector and I removed it, but the need has come back and I am unable to get paging to work.

The aggregation operation works just fine without paging, all the attributes are mapped for the first block of data that is returned.

When I add the paging code, I get an exception. This is the exact code I and others in this forum have previously used. Anyone know if something changed recently?

The first line of the exception is pretty clear (I assume the message was truncated by the exception), what is needed to change the type from a string to what $endpoint.fullUrl$ requires?

TERMINATE_IF $response.['@odata.nextLink']$ == NULL
$endpoint.fullUrl$ = $response.['@odata.nextLink']$

java.lang.RuntimeException - java.lang.IllegalStateException: [ConnectorError] Unknown value type odata.nextLi (requestId: 2a88b9484785468f82ccb6dbe17cce37) - java.lang.RuntimeException: java.lang.IllegalStateException: [ConnectorError] Unknown value type odata.nextLi (requestId: 2a88b9484785468f82ccb6dbe17cce37)
at com.sailpoint.mantis.qpoc.message.AccountAggregation.iterateResourceObjects_aroundBody6(AccountAggregation.java:659)
at com.sailpoint.mantis.qpoc.message.AccountAggregation$AjcClosure7.run(AccountAggregation.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:179)
at com.sailpoint.tracing.otel.TracedAspect.lambda$traceExecution$0(TracedAspect.java:38)
at com.sailpoint.tracing.otel.GlobalTracer.trace(GlobalTracer.java:170)
at com.sailpoint.tracing.otel.GlobalTracer.trace(GlobalTracer.java:143)
at com.sailpoint.tracing.otel.TracedAspect.traceExecution(TracedAspect.java:40)
at com.sailpoint.mantis.qpoc.message.AccountAggregation.iterateResourceObjects(AccountAggregation.java:592)
at com.sailpoint.mantis.qpoc.message.AccountAggregation.handleMessage_aroundBody0(AccountAggregation.java:359)
at com.sailpoint.mantis.qpoc.message.AccountAggregation$AjcClosure1.run(AccountAggregation.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:179)
at com.sailpoint.atlas.metrics.MessageMetricsAspect.meterMessageTimeAndExceptions(MessageMetricsAspect.java:100)
at com.sailpoint.mantis.qpoc.message.AccountAggregation.handleMessage(AccountAggregation.java:340)
at com.sailpoint.atlas.messaging.server.TypeMessageHandler.handleMessage(TypeMessageHandler.java:87)
at com.sailpoint.mantis.qpoc.utility.QpocMessageHandler.handleMessage_aroundBody0(QpocMessageHandler.java:60)
at com.sailpoint.mantis.qpoc.utility.QpocMessageHandler$AjcClosure1.run(QpocMessageHandler.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:179)
at com.sailpoint.atlas.metrics.MessageMetricsAspect.meterMessageTimeAndExceptions(MessageMetricsAspect.java:100)
at com.sailpoint.mantis.qpoc.utility.QpocMessageHandler.handleMessage(QpocMessageHandler.java:52)
at com.sailpoint.mantis.platform.message.ObjectConfigMessageHandler.handleMessage(ObjectConfigMessageHandler.java:33)
at com.sailpoint.atlas.tracing.plugin.otel.TraceMessageHandler.lambda$handleMessage$0(TraceMessageHandler.java:65)
at com.sailpoint.atlas.tracing.otel.Trace.trace(Trace.java:75)
at com.sailpoint.atlas.tracing.plugin.otel.TraceMessageHandler.handleMessage(TraceMessageHandler.java:60)
at com.sailpoint.atlas.message.DynamicMessageHandler$ChainedMessageHandlerAdapter.handleMessage(DynamicMessageHandler.java:56)
at com.sailpoint.atlas.tracing.plugin.TracingMessageHandler.handleMessage(TracingMessageHandler.java:88)
at com.sailpoint.atlas.message.DynamicMessageHandler$ChainedMessageHandlerAdapter.handleMessage(DynamicMessageHandler.java:56)
at com.sailpoint.atlas.usage.plugin.UsageMessageHandler.handleMessage(UsageMessageHandler.java:36)
at com.sailpoint.atlas.message.DynamicMessageHandler$ChainedMessageHandlerAdapter.handleMessage(DynamicMessageHandler.java:56)
at com.sailpoint.atlas.message.DynamicMessageHandler.handleMessage(DynamicMessageHandler.java:37)
at com.sailpoint.mantis.platform.message.SailPointContextMessageHandler.handleMessage(SailPointContextMessageHandler.java:55)
at com.sailpoint.atlas.message.FailureNotificationHandler.handleMessage(FailureNotificationHandler.java:58)
at com.sailpoint.atlas.message.RequestContextMessageHandler.handleMessage(RequestContextMessageHandler.java:77)
at com.sailpoint.mantis.platform.message.ExceptionMessageHandler.handleMessage(ExceptionMessageHandler.java:49)
at com.sailpoint.atlas.messaging.server.MessageProcessor.handleJobMessage(MessageProcessor.java:278)
at com.sailpoint.atlas.messaging.server.MessageProcessor.handleMessage(MessageProcessor.java:140)
at com.sailpoint.atlas.messaging.server.MessageProcessor.lambda$asyncHandleMessage$0(MessageProcessor.java:112)
at com.sailpoint.atlas.messaging.server.MessageProcessor.withOrgLogging(MessageProcessor.java:197)
at com.sailpoint.atlas.messaging.server.MessageProcessor.withReportingAndOrgLogging(MessageProcessor.java:187)
at com.sailpoint.atlas.messaging.server.MessageProcessor.lambda$asyncHandleMessage$1(MessageProcessor.java:112)
at com.sailpoint.atlas.messaging.server.impl.SourceRunnableImpl.run(SourceRunnableImpl.java:77)
at com.sailpoint.atlas.messaging.server.impl.BufferedSourceQueue$IncrementingSourceRunnable.run(BufferedSourceQueue.java:181)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: [ConnectorError] Unknown value type odata.nextLi (requestId: 2a88b9484785468f82ccb6dbe17cce37)
at com.sailpoint.connector.cloud.spconnect.SpConnectProxy$1.nextResponse_aroundBody0(SpConnectProxy.java:417)
at com.sailpoint.connector.cloud.spconnect.SpConnectProxy$1$AjcClosure1.run(SpConnectProxy.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:179)
at com.sailpoint.tracing.otel.TracedAspect.lambda$traceExecution$0(TracedAspect.java:38)
at com.sailpoint.tracing.otel.GlobalTracer.trace(GlobalTracer.java:170)
at com.sailpoint.tracing.otel.GlobalTracer.trace(GlobalTracer.java:143)
at com.sailpoint.tracing.otel.TracedAspect.traceExecution(TracedAspect.java:40)
at com.sailpoint.connector.cloud.spconnect.SpConnectProxy$1.nextResponse(SpConnectProxy.java:401)
at sailpoint.connector.cloud.CloudConnector$CloudBridgeIterator.buildDataBlockIterator(CloudConnector.java:1243)
at sailpoint.connector.cloud.CloudConnector$CloudBridgeIterator.checkForMoreData(CloudConnector.java:1231)
at sailpoint.connector.cloud.CloudConnector$CloudBridgeIterator.hasNext(CloudConnector.java:1157)
at sailpoint.connector.ConnectorProxy$CustomizingIterator.peek(ConnectorProxy.java:714)
at sailpoint.connector.ConnectorProxy$CustomizingIterator.hasNext(ConnectorProxy.java:741)
at com.sailpoint.mantis.qpoc.message.AccountAggregation.iterateResourceObjects_aroundBody6(AccountAggregation.java:627)
... 45 more

MVKR7T · November 11, 2024, 7:54pm

Can you paste the sample response from your API here

liamkokeeffe · November 11, 2024, 8:47pm

Hey Joseph,

I think the special characters are causing issues with the velocity code. Can you try the below and let me know if it works?

#set( $a = "@")
#set( $p = ".")
#set($url = "${a}odata${p}nextLink")
TERMINATE_IF $response[$url]$ == NULL
$endpoint.fullUrl$ = $response[$url]$

Thanks,

Liam

josephcasale · November 12, 2024, 11:49am

Hi @MVKR7T ,
The response looks like this: List users - Microsoft Graph v1.0 | Microsoft Learn

The specific link looks like this: Paging Microsoft Graph data in your app - Microsoft Graph | Microsoft Learn

josephcasale · November 12, 2024, 11:58am

Hi @liamkokeeffe,
That paging configuration resulting in one additional try, I see 1998 users were fetched.

Is it possible to log the nexLink in the SaaS connector and obtain the results?

liamkokeeffe · November 12, 2024, 3:26pm

Hey Joseph,

How many users are being returned per page? At a glance, the 1998 total makes me think it reached the last page (i.e if the page total is 50/100/1000).

Thanks,

Liam

josephcasale · November 12, 2024, 3:43pm

Hi @liamkokeeffe
I am returning 999 users per page, so 1998 indicates the connector goes through two iterations. There are about 20,000 users that should be returned.

When I execute the query from the command line and manually implement a loop, it works as expected.

liamkokeeffe · November 12, 2024, 7:57pm

Interesting, not sure what the issue is. I’m not aware of being able to log the nextLink url through the paging steps.

As an alternative, you could setup before/after operation rules to handle the paging and/or log the nextLink url that is being returned. I’ve attached documentation on using operation rules to handle paging - Before and After Operation Rules Configuration

Thanks,

Liam

josephcasale · November 13, 2024, 9:58pm

I opened an AAA session and the architect located an open issue on this topic.

The VTL parsing behavior for Web Services connectors differs between non-SaaS and SaaS configurations. In the SaaS connector, the @ symbol must be escaped with a ` (grave accent) symbol.

The correct graph paging configuration for non-SaaS connectors is:

TERMINATE_IF $response.['@odata.nextLink']$ == NULL
$endpoint.fullUrl$ = $response.['@odata.nextLink']$

The correct graph paging configuration for SaaS connectors is:

TERMINATE_IF $response.['`@odata.nextLink']$ == NULL
$endpoint.fullUrl$ = $response.['`@odata.nextLink']$

I hope that helps someone!

system · January 12, 2025, 9:58pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Custom Web Services Connector Pagination failing ISC Discussion and Questions webservice-connector , aggregation , identity-security-cloud	10	250	July 28, 2024
Pagination not working on web service connector ISC Discussion and Questions webservice-connector , aggregation , identity-security-cloud	9	101	December 13, 2024
Paging issue in webservices connector ISC Discussion and Questions webservice-connector , identity-security-cloud	7	344	June 16, 2024
Web Service Connector not Aggregating all Accounts ISC Discussion and Questions identity-security-cloud	3	584	February 9, 2024
NERM Web Services SAAS Connector Configuration - Paging ISC Discussion and Questions webservice-connector , apis , aggregation , identity-security-cloud , nerm	4	300	July 12, 2024

Web services paging problem

Related topics