Exception in CCG logs: java.lang.IllegalStateException\n\tat com.sailpoint.pipeline.util.Decrypter.decryptObject

Identity Security Cloud (ISC) ISC Discussion and Questions
,
1

Hi Experts,

We see the below exceptions in the ccg logs, would anyone know what could cause this?



{"exception":{"stacktrace":"java.lang.IllegalStateException\n\tat com.sailpoint.pipeline.util.Decrypter.decryptObject(Decrypter.java:70)\n\tat com.sailpoint.pipeline.server.PipelineServer$InboundQueueListener.handleDecryption(PipelineServer.java:342)\n\tat com.sailpoint.pipeline.server.PipelineServer$InboundQueueListener.onMes
sage(PipelineServer.java:300)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage_aroundBody0(AbstractBaseQueue.java:81)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer$AjcClosure1.run(AbstractBaseQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl
.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.TimedAspect.logTimed(TimedAspect.java:24)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage(AbstractBaseQueue.java:79)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run_aroundBody0(AbstractSQ
SQueue.java:237)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer$AjcClosure1.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect.java:46)\n\tat com.sailpoint.pip
eline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run(AbstractSQSQueue.java:208)\n\tat java.base\/java.lang.Thread.run(Thread.java:829)\n","exception_class":"java.lang.IllegalStateException"},"stack":"ccg","pod":"stg03-eucentral1","connector-logging":"163","clusterId":"205","utilities":"1.8.7","buildNumber":"1008","apiUs
ername":"d44ee362-61ad-4f56-90c5-5d21ed13b668","orgType":"","file":"Decrypter.java","encryption":"1.8.7","connector-bundle-identityiq":"233","line_number":85,"@version":1,"cloud-modules-api":"1477","logger_name":"com.sailpoint.pipeline.util.Decrypter","mantis-client":"1.8.7","class":"com.sailpoint.pipeline.util.Decrypter","atl
as-api":"1823","va-gateway-client":"52","tracing":"1.8.7","clientId":"d44ee362-61ad-4f56-90c5-5d21ed13b668","source_host":"b6a2d1281837","method":"decryptObject","org":"XXXXXXX-sb","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"An error occurred while decrypting the message.","pipeline":"1.8.
7","@timestamp":"2024-08-16T07:28:53.486Z","thread_name":"Thread-1","atlas-util":"1823","metrics":"1.8.7","region":"eu-central-1","queue":"stg03-eucentral1-XXXXXXX-sb-cluster-78d773edb4bb","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"exception":{"stacktrace":"java.lang.IllegalStateException: java.lang.IllegalStateException\n\tat com.sailpoint.pipeline.util.Decrypter.decryptObject(Decrypter.java:86)\n\tat com.sailpoint.pipeline.server.PipelineServer$InboundQueueListener.handleDecryption(PipelineServer.java:342)\n\tat com.sailpoint.pipeline.server.Pipeline
Server$InboundQueueListener.onMessage(PipelineServer.java:300)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage_aroundBody0(AbstractBaseQueue.java:81)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer$AjcClosure1.run(AbstractBaseQueue.java:1)\n\tat org.aspe
ctj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.TimedAspect.logTimed(TimedAspect.java:24)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage(AbstractBaseQueue.java:79)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageCo
nsumer.run_aroundBody0(AbstractSQSQueue.java:237)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer$AjcClosure1.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect
.java:46)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run(AbstractSQSQueue.java:208)\n\tat java.base\/java.lang.Thread.run(Thread.java:829)\nCaused by: java.lang.IllegalStateException\n\tat com.sailpoint.pipeline.util.Decrypter.decryptObject(Decrypter.java:70)\n\t... 13 more\n","exception_cla
ss":"java.lang.IllegalStateException","exception_message":"java.lang.IllegalStateException"},"stack":"ccg","pod":"stg03-eucentral1","connector-logging":"163","clusterId":"205","utilities":"1.8.7","buildNumber":"1008","apiUsername":"d44ee362-61ad-4f56-90c5-5d21ed13b668","orgType":"","file":"AbstractBaseQueue.java","encryption":
"1.8.7","connector-bundle-identityiq":"233","line_number":84,"@version":1,"cloud-modules-api":"1477","logger_name":"com.sailpoint.pipeline.queue.AbstractBaseQueue","mantis-client":"1.8.7","class":"com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer","atlas-api":"1823","va-gateway-client":"52","tracing":"1.8.
7","clientId":"d44ee362-61ad-4f56-90c5-5d21ed13b668","source_host":"b6a2d1281837","method":"dispatchMessage_aroundBody0","org":"XXXXXXX-sb","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"An error occurred processing a message.","pipeline":"1.8.7","@timestamp":"2024-08-16T07:28:53.486Z","threa
d_name":"Thread-1","atlas-util":"1823","metrics":"1.8.7","region":"eu-central-1","queue":"stg03-eucentral1-XXXXXXX-sb-cluster-78d773edb4bb","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
dup {"stack":"ccg","pod":"stg03-eucentral1","connector-logging":"163","clusterId":"205","utilities":"1.8.7","buildNumber":"1008","apiUsername":"d44ee362-61ad-4f56-90c5-5d21ed13b668","orgType":"","file":"Decrypter.java","encryption":"1.8.7","connector-bundle-identityiq":"233","line_number":69,"@version":1,"cloud-modules-api":"1
477","logger_name":"com.sailpoint.pipeline.util.Decrypter","mantis-client":"1.8.7","class":"com.sailpoint.pipeline.util.Decrypter","atlas-api":"1823","va-gateway-client":"52","tracing":"1.8.7","clientId":"d44ee362-61ad-4f56-90c5-5d21ed13b668","source_host":"b6a2d1281837","method":"decryptObject","org":"XXXXXXX-sb","level":"ERR
OR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Cannot find key to decrypt message","pipeline":"1.8.7","@timestamp":"2024-08-16T07:29:33.044Z","thread_name":"Thread-1","atlas-util":"1823","metrics":"1.8.7","region":"eu-central-1","queue":"stg03-eucentral1-XXXXXXX-sb-cluster-78d773edb4bb","SCIM Common":"8.
0 Build 00b1f252d1b-20200225-190809"}
{"exception":{"stacktrace":"java.lang.IllegalStateException\n\tat com.sailpoint.pipeline.util.Decrypter.decryptObject(Decrypter.java:70)\n\tat com.sailpoint.pipeline.server.PipelineServer$InboundQueueListener.handleDecryption(PipelineServer.java:342)\n\tat com.sailpoint.pipeline.server.PipelineServer$InboundQueueListener.onMes
sage(PipelineServer.java:300)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage_aroundBody0(AbstractBaseQueue.java:81)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer$AjcClosure1.run(AbstractBaseQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl
.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.TimedAspect.logTimed(TimedAspect.java:24)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage(AbstractBaseQueue.java:79)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run_aroundBody0(AbstractSQ
SQueue.java:237)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer$AjcClosure1.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect.java:46)\n\tat com.sailpoint.pip
eline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run(AbstractSQSQueue.java:208)\n\tat java.base\/java.lang.Thread.run(Thread.java:829)\n","exception_class":"java.lang.IllegalStateException"},"stack":"ccg","pod":"stg03-eucentral1","connector-logging":"163","clusterId":"205","utilities":"1.8.7","buildNumber":"1008","apiUs
ername":"d44ee362-61ad-4f56-90c5-5d21ed13b668","orgType":"","file":"Decrypter.java","encryption":"1.8.7","connector-bundle-identityiq":"233","line_number":85,"@version":1,"cloud-modules-api":"1477","logger_name":"com.sailpoint.pipeline.util.Decrypter","mantis-client":"1.8.7","class":"com.sailpoint.pipeline.util.Decrypter","atl
as-api":"1823","va-gateway-client":"52","tracing":"1.8.7","clientId":"d44ee362-61ad-4f56-90c5-5d21ed13b668","source_host":"b6a2d1281837","method":"decryptObject","org":"XXXXXXX-sb","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"An error occurred while decrypting the message.","pipeline":"1.8.
7","@timestamp":"2024-08-16T07:29:33.044Z","thread_name":"Thread-1","atlas-util":"1823","metrics":"1.8.7","region":"eu-central-1","queue":"stg03-eucentral1-XXXXXXX-sb-cluster-78d773edb4bb","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"exception":{"stacktrace":"java.lang.IllegalStateException: java.lang.IllegalStateException\n\tat com.sailpoint.pipeline.util.Decrypter.decryptObject(Decrypter.java:86)\n\tat com.sailpoint.pipeline.server.PipelineServer$InboundQueueListener.handleDecryption(PipelineServer.java:342)\n\tat com.sailpoint.pipeline.server.Pipeline
Server$InboundQueueListener.onMessage(PipelineServer.java:300)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage_aroundBody0(AbstractBaseQueue.java:81)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer$AjcClosure1.run(AbstractBaseQueue.java:1)\n\tat org.aspe
ctj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.TimedAspect.logTimed(TimedAspect.java:24)\n\tat com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer.dispatchMessage(AbstractBaseQueue.java:79)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageCo
nsumer.run_aroundBody0(AbstractSQSQueue.java:237)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer$AjcClosure1.run(AbstractSQSQueue.java:1)\n\tat org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:167)\n\tat com.sailpoint.metrics.HistogramTimedAspect.logTimed(HistogramTimedAspect
.java:46)\n\tat com.sailpoint.pipeline.queue.AbstractSQSQueue$AbstractSQSMessageConsumer.run(AbstractSQSQueue.java:208)\n\tat java.base\/java.lang.Thread.run(Thread.java:829)\nCaused by: java.lang.IllegalStateException\n\tat com.sailpoint.pipeline.util.Decrypter.decryptObject(Decrypter.java:70)\n\t... 13 more\n","exception_cla
ss":"java.lang.IllegalStateException","exception_message":"java.lang.IllegalStateException"},"stack":"ccg","pod":"stg03-eucentral1","connector-logging":"163","clusterId":"205","utilities":"1.8.7","buildNumber":"1008","apiUsername":"d44ee362-61ad-4f56-90c5-5d21ed13b668","orgType":"","file":"AbstractBaseQueue.java","encryption":
"1.8.7","connector-bundle-identityiq":"233","line_number":84,"@version":1,"cloud-modules-api":"1477","logger_name":"com.sailpoint.pipeline.queue.AbstractBaseQueue","mantis-client":"1.8.7","class":"com.sailpoint.pipeline.queue.AbstractBaseQueue$AbstractMessageConsumer","atlas-api":"1823","va-gateway-client":"52","tracing":"1.8.
7","clientId":"d44ee362-61ad-4f56-90c5-5d21ed13b668","source_host":"b6a2d1281837","method":"dispatchMessage_aroundBody0","org":"XXXXXXX-sb","level":"ERROR","IdentityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"An error occurred processing a message.","pipeline":"1.8.7","@timestamp":"2024-08-16T07:29:33.044Z","threa
d_name":"Thread-1","atlas-util":"1823","metrics":"1.8.7","region":"eu-central-1","queue":"stg03-eucentral1-XXXXXXX-sb-cluster-78d773edb4bb","SCIM Common":"8.0 Build 00b1f252d1b-20200225-190809"}
{"stack":"ccg","pod":"stg03-eucentral1","connector-logging":"163","clusterId":"205","utilities":"1.8.7","buildNumber":"1008","apiUsername":"d44ee362-61ad-4f56-90c5-5d21ed13b668","orgType":"","file":"PollService.java","encryption":"1.8.7","connector-bundle-identityiq":"233","line_number":250,"@version":1,"cloud-modules-api":"14
77","logger_name":"sailpoint.gateway.service.PollService","mantis-client":"1.8.7","class":"sailpoint.gateway.service.PollService","atlas-api":"1823","va-gateway-client":"52","tracing":"1.8.7","clientId":"d44ee362-61ad-4f56-90c5-5d21ed13b668","source_host":"b6a2d1281837","method":"pollCegs","org":"XXXXXXX-sb","level":"INFO","Id
entityIQ":"8.3p4 Build 1527a593753-20230805-223436","message":"Polling CEGS","pipeline":"1.8.7","@timestamp":"2024-08-16T07:29:52.094Z","thread_name":"Thread-3","atlas-util":"1823","metrics":"1.8.7","region":"eu-central-1","queue":"stg03-eucentral1-XXXXXXX-sb-cluster-78d773edb4bb","SCIM Common":"8.0 Build 00b1f252d1b-20200225-
190809"}

Also, test connectivity to our targets is unstable. It works only some times, and most of the times, it fails with a timeout error. And when it fails, we see the above exceptions in the logs. Time is already in sync for both the VAs.

2

Hi @tamalika01,

Refer the below link. It will be helpful.
https://community.sailpoint.com/t5/IdentityNow-Forum/Sailpoint-IDN-VA-AWS-getting-error-on-source-test-connection/m-p/217833

Regards,
Arun

2 Likes
3

hey @tamalika01 this is problems with the VA cluster Keys.

Create another cluster in ISC and reconfigure the VA in it.

This is gonna sovle your issue.

1 Like
4

@tamalika01 you can add some some time so that timeout does not occur.
Also raise a ticket with sailpoint sometimes its an issues with the ccg version of sailpoint which may cause such issues.
try to check if you ccg is up and running and stable for 5 minutes using the command
sudo docker ps

1 Like
5

@schattopadhy thanks we already raised this with Sailpoint and we receive another VA image from them. We are yet to use it. CCG is consistently up since a couple of days already. We are worried that this does not happen with the next VA we configure.

6

@ipobeidi we have 2 VAs in the cluster and we see this exception periodically in both the VAs. What could be the pßroblem with the cluster keys? Because the VA cluster is always healthy in the UI.
And by cluster keys, you mean the credentials we upload in the config.yaml file, right?

7

No, the cluster itself have a set of keys to encrypt the messages, so only the va within those cluster can decrypt the messages.

Sometimes the key get mess up and you need to rebuild it.
Also is good to remember that the passphrase on the yaml file need to be the same for any va on the cluster.

1 Like
8

Hi @tamalika01 , It would be beneficial to Deploy High Availability and Disaster Recovery for your VAs in production in case it happens again
Deploying Virtual Appliances - SailPoint Identity Services

1 Like
9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.