We are attempting to test connection from our Virtual Appliance deployed in customer’s commercial cloud environment to a target system placed in the same cloud environment. This is configured using the web services connector as the target system is exposing API’s and will act as an authoritative source. However, when we test connection on IdentityNow UI , we get hit with a timeout error.
After checking the ccg logs, we observe that the error is related to amazonSQS connectivity :
>{"exception":{"stacktrace":"java.io.IOException: Server returned HTTP response code: 403 for URL: http:\/\/www.w3.org\/TR\/html4\/strict.dtd\n\tat java.base\/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)\n\tat java.base\/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:676)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1396)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1362)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:257)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1152)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1040)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:943)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)\n\tat java.xml\/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534)\n\tat java.xml\/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)\n\tat java.xml\/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)\n\tat java.xml\/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)\n\tat java.xml\/com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:246)\n\tat java.xml\/com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)\n\tat java.xml\/javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:122)\n\tat com.amazonaws.util.XpathUtils.documentFrom(XpathUtils.java:172)\n\tat com.amazonaws.util.XpathUtils.documentFrom(XpathUtils.java:179)\n\tat com.amazonaws.http.DefaultErrorResponseHandler.parseXml(DefaultErrorResponseHandler.java:124)\n\tat com.amazonaws.http.DefaultErrorResponseHandler.documentFromContent(DefaultErrorResponseHandler.java:105)\n\tat com.amazonaws.http.DefaultErrorResponseHandler.createAse(DefaultErrorResponseHandler.java:84)\n\tat com.amazonaws.http.DefaultErrorResponseHandler.handle(DefaultErrorResponseHandler.java:71)\n\tat com.amazonaws.http.DefaultErrorResponseHandler.handle(DefaultErrorResponseHandler.java:47)\n\tat com.amazonaws.http.AwsErrorResponseHandler.handleAse(AwsErrorResponseHandler.java:50)\n\tat com.amazonaws.http.AwsErrorResponseHandler.handle(AwsErrorResponseHandler.java:38)\n\tat com.amazonaws.http.AwsErrorResponseHandler.handle(AwsErrorResponseHandler.java:24)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1622)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2033)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2009)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1084)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1060)\n\tat com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:260)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl_aroundBody2(AbstractSQSQueue.java:112)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AjcClosure3.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect.java:46)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl(AbstractSQSQueue.java:104)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.isHealthy(AbstractSQSQueue.java:153)\n\tat com.sailpoint.pipeline.server.PipelineServer.isHealthy(PipelineServer.java:165)\n\tat sailpoint.gateway.service.PipelineService.start(PipelineService.java:197)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.lambda$0(PrioritizedServiceLifecycler.java:35)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)\n\tat java.base\/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)\n\tat java.base\/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)\n\tat java.base\/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)\n\tat java.base\/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.base\/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.startupServices(PrioritizedServiceLifecycler.java:33)\n\tat sailpoint.startup.Main.startCcg(Main.java:182)\n\tat sailpoint.startup.Main.main(Main.java:77)\n","exception_class":"java.io.IOException","exception_message":"Server returned HTTP response code: 403 for URL: http:\/\/www.w3.org\/TR\/html4\/strict.dtd"},"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","clusterId":"17","buildNumber":"912","apiUsername":"7bf66f74-9acd-413e-92aa-69031abbc417","orgType":"","file":"DefaultErrorResponseHandler.java","encryption":"1266","connector-bundle-identityiq":"202","line_number":126,"@version":1,"logger_name":"com.amazonaws.http.DefaultErrorResponseHandler","mantis-client":"1266","class":"com.amazonaws.http.DefaultErrorResponseHandler","atlas-api":"1752","va-gateway-client":"40","clientId":"84","source_host":"2fbd3a6c9233","method":"parseXml","org":"aiam-stg","level":"DEBUG","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Unable to parse HTTP response (Invocation Id:74851075-0b71-c222-f1ab-a37c1bb1f9ae) content to XML document '<!DOCTYPE html PUBLIC \"-\/\/W3C\/\/DTD HTML 4.01\/\/EN\" \"http:\/\/www.w3.org\/TR\/html4\/strict.dtd\">\n<html><head>\n<meta type=\"copyright\" content=\"Copyright (C) 1996-2021 The Squid Software Foundation and contributors\">\n<meta http-equiv=\"Content-Type\" content=\"text\/html; charset=utf-8\">\n<title>ERROR: The requested URL could not be retrieved<\/title>\n<style type=\"text\/css\"><!--\n \/*\n * Copyright (C) 1996-2021 The Squid Software Foundation and contributors\n *\n * Squid software is distributed under GPLv2+ license and includes\n * contributions from numerous individuals and organizations.\n * Please see the COPYING and CONTRIBUTORS files for details.\n *\/\n\n\/*\n Stylesheet for Squid Error pages\n Adapted from design by Free CSS Templates\n http:\/\/www.freecsstemplates.org\n Released for free under a Creative Commons Attribution 2.5 License\n*\/\n\n\/* Page basics *\/\n* {\n\tfont-family: verdana, sans-serif;\n}\n\nhtml body {\n\tmargin: 0;\n\tpadding: 0;\n\tbackground: #efefef;\n\tfont-size: 12px;\n\tcolor: #1e1e1e;\n}\n\n\/* Page displayed title area *\/\n#titles {\n\tmargin-left: 15px;\n\tpadding: 10px;\n\tpadding-left: 100px;\n\tbackground: url('\/squid-internal-static\/icons\/SN.png') no-repeat left;\n}\n\n\/* initial title *\/\n#titles h1 {\n\tcolor: #000000;\n}\n#titles h2 {\n\tcolor: #000000;\n}\n\n\/* special event: FTP success page titles *\/\n#titles ftpsuccess {\n\tbackground-color:#00ff00;\n\twidth:100%;\n}\n\n\/* Page displayed body content area *\/\n#content {\n\tpadding: 10px;\n\tbackground: #ffffff;\n}\n\n\/* General text *\/\np {\n}\n\n\/* error brief description *\/\n#error p {\n}\n\n\/* some data which may have caused the problem *\/\n#data {\n}\n\n\/* the error message received from the system or other software *\/\n#sysmsg {\n}\n\npre {\n}\n\n\/* special event: FTP \/ Gopher directory listing *\/\n#dirmsg {\n font-family: courier, monospace;\n color: black;\n font-size: 10pt;\n}\n#dirlisting {\n margin-left: 2%;\n margin-right: 2%;\n}\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\n border-bottom: groove;\n}\n#dirlisting td.size {\n width: 50px;\n text-align: right;\n padding-right: 5px;\n}\n\n\/* horizontal lines *\/\nhr {\n\tmargin: 0;\n}\n\n\/* page displayed footer area *\/\n#footer {\n\tfont-size: 9px;\n\tpadding-left: 10px;\n}\n\n\nbody\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\n:lang(he) { direction: rtl; }\n --><\/style>\n<\/head><body id=ERR_ACCESS_DENIED>\n<div id=\"titles\">\n<h1>ERROR<\/h1>\n<h2>The requested URL could not be retrieved<\/h2>\n<\/div>\n<hr>\n\n<div id=\"content\">\n<p>The following error was encountered while trying to retrieve the URL: <a href=\"https:\/\/sqs.ap-southeast-1.amazonaws.com\/*\">https:\/\/sqs.ap-southeast-1.amazonaws.com\/*<\/a><\/p>\n\n<blockquote id=\"error\">\n<p><b>Access Denied.<\/b><\/p>\n<\/blockquote>\n\n<p>Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.<\/p>\n\n<p>Your cache administrator is <a href=\"mailto:root?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20parent_squid%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Thu,%2024%20Aug%202023%2007%3A03%3A46%20GMT%0D%0A%0D%0AClientIP%3A%2010.53.0.73%0D%0A%0D%0AHTTP%20Request%3A%0D%0ACONNECT%20%20HTTP%2F1.1%0AUser-Agent%3A%20Apache-HttpClient%2F4.5.13%20(Java%2F11.0.20)%0D%0AHost%3A%20sqs.ap-southeast-1.amazonaws.com%3A443%0D%0A%0D%0A%0D%0A\">root<\/a>.<\/p>\n<br>\n<\/div>\n\n<hr>\n<div id=\"footer\">\n<p>Generated Thu, 24 Aug 2023 07:03:46 GMT by parent_squid (squid\/4.15)<\/p>\n<!-- ERR_ACCESS_DENIED -->\n<\/div>\n<\/body><\/html>\n' ","pipeline":"1266","@timestamp":"2023-08-24T07:03:18.662Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-17","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","clusterId":"17","buildNumber":"912","apiUsername":"7bf66f74-9acd-413e-92aa-69031abbc417","orgType":"","file":"AmazonHttpClient.java","encryption":"1266","connector-bundle-identityiq":"202","line_number":1624,"@version":1,"logger_name":"com.amazonaws.request","mantis-client":"1266","class":"com.amazonaws.http.AmazonHttpClient$RequestExecutor","atlas-api":"1752","va-gateway-client":"40","clientId":"84","source_host":"2fbd3a6c9233","method":"handleErrorResponse","org":"aiam-stg","level":"DEBUG","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Received error response: com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 403; Error Code: 403 Forbidden; Request ID: null)","pipeline":"1266","@timestamp":"2023-08-24T07:03:18.710Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-17","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"exception":{"stacktrace":"com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 403; Error Code: 403 Forbidden; Request ID: null)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2033)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2009)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1084)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1060)\n\tat com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:260)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl_aroundBody2(AbstractSQSQueue.java:112)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AjcClosure3.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect.java:46)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl(AbstractSQSQueue.java:104)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.isHealthy(AbstractSQSQueue.java:153)\n\tat com.sailpoint.pipeline.server.PipelineServer.isHealthy(PipelineServer.java:165)\n\tat sailpoint.gateway.service.PipelineService.start(PipelineService.java:197)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.lambda$0(PrioritizedServiceLifecycler.java:35)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)\n\tat java.base\/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)\n\tat java.base\/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)\n\tat java.base\/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)\n\tat java.base\/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.base\/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.startupServices(PrioritizedServiceLifecycler.java:33)\n\tat sailpoint.startup.Main.startCcg(Main.java:182)\n\tat sailpoint.startup.Main.main(Main.java:77)\n","exception_class":"com.amazonaws.services.sqs.model.AmazonSQSException","exception_message":"null (Service: AmazonSQS; Status Code: 403; Error Code: 403 Forbidden; Request ID: null)"},"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","clusterId":"17","buildNumber":"912","apiUsername":"7bf66f74-9acd-413e-92aa-69031abbc417","orgType":"","file":"AbstractSQSQueue.java","encryption":"1266","connector-bundle-identityiq":"202","line_number":120,"@version":1,"logger_name":"com.sailpoint.pipeline.queue.AbstractSQSQueue","mantis-client":"1266","class":"com.sailpoint.pipeline.queue.AbstractSQSQueue","atlas-api":"1752","va-gateway-client":"40","clientId":"84","source_host":"dc15da64c7b9","method":"getQueueUrl_aroundBody2","org":"aiam-stg","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Unable to determine SQS queue url.","pipeline":"1266","@timestamp":"2023-08-23T19:03:18.114Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-17","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
> {"exception":{"stacktrace":"com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 403; Error Code: 403 Forbidden; Request ID: null)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2033)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2009)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1084)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1060)\n\tat com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:260)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl_aroundBody2(AbstractSQSQueue.java:112)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AjcClosure3.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect.java:46)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl(AbstractSQSQueue.java:104)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run_aroundBody0(AbstractSQSQueue.java:211)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer$AjcClosure1.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect.java:46)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run(AbstractSQSQueue.java:208)\n\tat java.base\/java.lang.Thread.run(Thread.java:829)\n","exception_class":"com.amazonaws.services.sqs.model.AmazonSQSException","exception_message":"null (Service: AmazonSQS; Status Code: 403; Error Code: 403 Forbidden; Request ID: null)"},"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","clusterId":"17","buildNumber":"912","apiUsername":"7bf66f74-9acd-413e-92aa-69031abbc417","orgType":"","file":"AbstractSQSQueue.java","encryption":"1266","connector-bundle-identityiq":"202","line_number":120,"@version":1,"logger_name":"com.sailpoint.pipeline.queue.AbstractSQSQueue","mantis-client":"1266","class":"com.sailpoint.pipeline.queue.AbstractSQSQueue","atlas-api":"1752","va-gateway-client":"40","clientId":"84","source_host":"dc15da64c7b9","method":"getQueueUrl_aroundBody2","org":"aiam-stg","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Unable to determine SQS queue url.","pipeline":"1266","@timestamp":"2023-08-23T19:03:18.117Z","thread_name":"Thread-1","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-17","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
> {"exception":{"stacktrace":"com.sailpoint.pipeline.PipelineException: Unable to determine SQS queue url.\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl_aroundBody2(AbstractSQSQueue.java:121)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AjcClosure3.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect.java:46)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl(AbstractSQSQueue.java:104)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.isHealthy(AbstractSQSQueue.java:153)\n\tat com.sailpoint.pipeline.server.PipelineServer.isHealthy(PipelineServer.java:165)\n\tat sailpoint.gateway.service.PipelineService.start(PipelineService.java:197)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.lambda$0(PrioritizedServiceLifecycler.java:35)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)\n\tat java.base\/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)\n\tat java.base\/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)\n\tat java.base\/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)\n\tat java.base\/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.base\/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.startupServices(PrioritizedServiceLifecycler.java:33)\n\tat sailpoint.startup.Main.startCcg(Main.java:182)\n\tat sailpoint.startup.Main.main(Main.java:77)\nCaused by: com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 403; Error Code: 403 Forbidden; Request ID: null)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2033)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2009)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1084)\n\tat com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1060)\n\tat com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:260)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue.getQueueUrl_aroundBody2(AbstractSQSQueue.java:112)\n\t... 19 more\n","exception_class":"com.sailpoint.pipeline.PipelineException","exception_message":"Unable to determine SQS queue url."},"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","clusterId":"17","buildNumber":"912","apiUsername":"7bf66f74-9acd-413e-92aa-69031abbc417","orgType":"","file":"AbstractSQSQueue.java","encryption":"1266","connector-bundle-identityiq":"202","line_number":157,"@version":1,"logger_name":"com.sailpoint.pipeline.queue.AbstractSQSQueue","mantis-client":"1266","class":"com.sailpoint.pipeline.queue.AbstractSQSQueue","atlas-api":"1752","va-gateway-client":"40","clientId":"84","source_host":"dc15da64c7b9","method":"isHealthy","org":"aiam-stg","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Healthcheck exception for queue: stg01-apsoutheast1-aiam-stg-cluster-17","pipeline":"1266","@timestamp":"2023-08-23T19:03:18.119Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-17","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
> {"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","clusterId":"17","buildNumber":"912","apiUsername":"7bf66f74-9acd-413e-92aa-69031abbc417","orgType":"","file":"AbstractSQSQueue.java","encryption":"1266","connector-bundle-identityiq":"202","line_number":163,"@version":1,"logger_name":"com.sailpoint.pipeline.queue.AbstractSQSQueue","mantis-client":"1266","class":"com.sailpoint.pipeline.queue.AbstractSQSQueue","atlas-api":"1752","va-gateway-client":"40","clientId":"84","source_host":"dc15da64c7b9","method":"isHealthy","org":"aiam-stg","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Queue 'stg01-apsoutheast1-aiam-stg-cluster-17' not healthy - credential class: com.sailpoint.pipeline.queue.DelegatedAmazonCredentials, hasCredentials: true, canConnect: false","pipeline":"1266","@timestamp":"2023-08-23T19:03:18.121Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-17","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"exception":{"stacktrace":"java.lang.RuntimeException: Failed starting pipeline server.\n\tat sailpoint.gateway.service.PipelineService.start(PipelineService.java:198)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.lambda$0(PrioritizedServiceLifecycler.java:35)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)\n\tat java.base\/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)\n\tat java.base\/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)\n\tat java.base\/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)\n\tat java.base\/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)\n\tat java.base\/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.base\/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)\n\tat sailpoint.gateway.service.impl.PrioritizedServiceLifecycler.startupServices(PrioritizedServiceLifecycler.java:33)\n\tat sailpoint.startup.Main.startCcg(Main.java:182)\n\tat sailpoint.startup.Main.main(Main.java:77)\n","exception_class":"java.lang.RuntimeException","exception_message":"Failed starting pipeline server."},"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","clusterId":"17","buildNumber":"912","apiUsername":"7bf66f74-9acd-413e-92aa-69031abbc417","orgType":"","file":"PipelineService.java","encryption":"1266","connector-bundle-identityiq":"202","line_number":203,"@version":1,"logger_name":"sailpoint.gateway.service.PipelineService","mantis-client":"1266","class":"sailpoint.gateway.service.PipelineService","atlas-api":"1752","va-gateway-client":"40","clientId":"84","source_host":"2fbd3a6c9233","method":"start","org":"aiam-stg","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Error occurred during Pipeline initialization.","pipeline":"1266","@timestamp":"2023-08-24T07:03:18.741Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-17","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","buildNumber":"913","apiUsername":"99b0760f-23f2-4dce-a968-efd8342ac9e3","orgType":"","file":"ResponseProcessCookies.java","encryption":"1266","connector-bundle-identityiq":"205","line_number":130,"@version":1,"logger_name":"org.apache.http.client.protocol.ResponseProcessCookies","mantis-client":"1266","class":"org.apache.http.client.protocol.ResponseProcessCookies","atlas-api":"1752","va-gateway-client":"45","source_host":"9b5fd1198dc8","method":"processCookies","org":"aiam-stg","level":"WARN","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Invalid cookie header: \"Set-Cookie: AWSALB=gmLXGDDzHSc37pJJLxauUAAWo6D31YnCWFzJDyCHwFH6XNtKMhVYtv5upUx4zan8Vh33KU\/nQPuMk\/3ScT\/QvEhF23eEHnn5awK51KzNipCx9lqcduAAQ+w4FhM8; Expires=Mon, 04 Sep 2023 19:03:18 GMT; Path=\/\". Invalid 'expires' attribute: Mon, 04 Sep 2023 19:03:18 GMT","pipeline":"1266","@timestamp":"2023-08-28T19:02:44.191Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-20","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"stack":"ccg","pod":"stg01-apsoutheast1","connector-logging":"148","buildNumber":"913","apiUsername":"99b0760f-23f2-4dce-a968-efd8342ac9e3","orgType":"","file":"ResponseProcessCookies.java","encryption":"1266","connector-bundle-identityiq":"205","line_number":130,"@version":1,"logger_name":"org.apache.http.client.protocol.ResponseProcessCookies","mantis-client":"1266","class":"org.apache.http.client.protocol.ResponseProcessCookies","atlas-api":"1752","va-gateway-client":"45","source_host":"9b5fd1198dc8","method":"processCookies","org":"aiam-stg","level":"WARN","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Invalid cookie header: \"Set-Cookie: AWSALBCORS=gmLXGDDzHSc37pJJLxauUAAWo6D31YnCWFzJDyCHwFH6XNtKMhVYtv5upUx4zan8Vh33KU\/nQPuMk\/3ScT\/QvEhF23eEHnn5awK51KzNipCx9lqcduAAQ+w4FhM8; Expires=Mon, 04 Sep 2023 19:03:18 GMT; Path=\/; SameSite=None; Secure\". Invalid 'expires' attribute: Mon, 04 Sep 2023 19:03:18 GMT","pipeline":"1266","@timestamp":"2023-08-28T19:02:44.192Z","thread_name":"main","atlas-util":"1752","metrics":"1266","region":"ap-southeast-1","queue":"stg01-apsoutheast1-aiam-stg-cluster-20","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
We have ensured that all the required whitelisting to amazon US-east as well as regional amazon servers has been done , and tested with curl commands. We are not sure how to debug this connectivity issue. Appreciate inputs from the community.
Thank you.