Seeing increased Hibernate session failures after upgrade to 8.3

[drosenbauer (Devin Rosenbauer)]

I’m seeing a lot of “failed to lazily initialize a collection of (whatever)” after the upgrade to 8.3. This happens, of course, if the object is detached from its Hibernate session. There is a consistent pattern of this within and after provisioning. In most cases, I can simply reload the Identity or Link objects in question after invoking the Provisioner. However, there are some cases that I don’t know how to handle neatly.

Here is an example stack trace from our customer’s installation (with the customer name edited out). IIQ was attempting to refresh the Identity during provisioning and could not load the referencedRules on a Business Role’s IdentitySelector Rule object.

2022-11-23T15:24:25,842 ERROR https-jsse-nio-8443-exec-4 iiq.plugins.match.ConflictResolutionService:518 - sailpoint.tools.GeneralException: failed to lazily initialize a collection of role: sailpoint.object.Rule.referencedRules, could not initialize proxy - no Session
sailpoint.tools.GeneralException: failed to lazily initialize a collection of role: sailpoint.object.Rule.referencedRules, could not initialize proxy - no Session
        at sailpoint.provisioning.IIQEvaluator.provision(IIQEvaluator.java:420) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.provisioning.PlanEvaluator.execute(PlanEvaluator.java:869) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.provisioning.PlanEvaluator.execute(PlanEvaluator.java:738) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.Provisioner.execute(Provisioner.java:1732) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.Identitizer.finishRefresh(Identitizer.java:2906) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.Identitizer.refresh(Identitizer.java:2491) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at example.customer.iiq.plugins.match.ResolveUtility.lambda$afterResolveCase$0(ResolveUtility.java:149) ~[?:?]
        at example.customer.iiq.plugins.match.resolver.CaseUtilities.handleCase(CaseUtilities.java:83) ~[?:?]
        at example.customer.iiq.plugins.match.ResolveUtility.afterResolveCase(ResolveUtility.java:64) ~[?:?]
        at example.customer.iiq.plugins.match.ResolveUtility.resolveCase(ResolveUtility.java:232) ~[?:?]
        at example.customer.iiq.plugins.match.ConflictResolutionService.lambda$resolveConflict$23(ConflictResolutionService.java:1152) ~[?:?]
        at com.identityworksllc.iiq.common.plugin.BaseCommonPluginResource.handle(BaseCommonPluginResource.java:587) ~[IIQCommon-plugins.jar:?]
        at com.identityworksllc.iiq.common.plugin.BaseCommonPluginResource.handle(BaseCommonPluginResource.java:650) ~[IIQCommon-plugins.jar:?]
        at example.customer.iiq.plugins.match.ConflictResolutionService.resolveConflict(ConflictResolutionService.java:1116) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [jersey-common-2.34.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [jersey-common-2.34.jar:?]
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [jersey-server-2.34.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) [jersey-container-servlet-core-2.34.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) [jersey-container-servlet-core-2.34.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) [jersey-container-servlet-core-2.34.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) [jersey-container-servlet-core-2.34.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) [jersey-container-servlet-core-2.34.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-websocket.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at sailpoint.web.SailPointResponseFilter.doFilter(SailPointResponseFilter.java:76) [identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at sailpoint.rest.jaxrs.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:90) [identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at sailpoint.rest.RestCsrfValidationFilter.doFilter(RestCsrfValidationFilter.java:71) [identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at sailpoint.rest.AuthenticationFilter.doFilter(AuthenticationFilter.java:109) [identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at sailpoint.web.SailPointContextRequestFilter.doFilter(SailPointContextRequestFilter.java:61) [identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at sailpoint.web.SailPointPollingRequestFilter.doFilter(SailPointPollingRequestFilter.java:151) [identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at sailpoint.web.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:63) [identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.20.RELEASE.jar:5.2.20.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.20.RELEASE.jar:5.2.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:9.0.41]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [catalina.jar:9.0.41]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [catalina.jar:9.0.41]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [catalina.jar:9.0.41]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [catalina.jar:9.0.41]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [catalina.jar:9.0.41]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) [catalina.jar:9.0.41]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [catalina.jar:9.0.41]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [catalina.jar:9.0.41]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-coyote.jar:9.0.41]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote.jar:9.0.41]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888) [tomcat-coyote.jar:9.0.41]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) [tomcat-coyote.jar:9.0.41]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:9.0.41]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:9.0.41]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: sailpoint.object.Rule.referencedRules, could not initialize proxy - no Session
        at org.hibernate.collection.internal.AbstractPersistentCollection.throwLazyInitializationException(AbstractPersistentCollection.java:606) ~[hibernate-core-5.4.27.Final.jar:5.4.27.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:218) ~[hibernate-core-5.4.27.Final.jar:5.4.27.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:585) ~[hibernate-core-5.4.27.Final.jar:5.4.27.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:149) ~[hibernate-core-5.4.27.Final.jar:5.4.27.Final]
        at org.hibernate.collection.internal.PersistentList.iterator(PersistentList.java:155) ~[hibernate-core-5.4.27.Final.jar:5.4.27.Final]
        at sailpoint.object.Rule.load(Rule.java:378) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.object.IdentitySelector.load(IdentitySelector.java:347) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.CorrelationModel$CorrelationRole.touch(CorrelationModel.java:1436) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.CorrelationModel$CorrelationRole.<init>(CorrelationModel.java:1428) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.CorrelationModel.addRole(CorrelationModel.java:1134) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.CorrelationModel.prepare(CorrelationModel.java:348) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.CorrelationModel.refresh(CorrelationModel.java:297) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.CorrelationModel.getCorrelationModel(CorrelationModel.java:258) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.EntitlementCorrelator.prepare(EntitlementCorrelator.java:542) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.api.EntitlementCorrelator.redetect(EntitlementCorrelator.java:1425) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.provisioning.IIQEvaluator.reDetectRoles(IIQEvaluator.java:832) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        at sailpoint.provisioning.IIQEvaluator.provision(IIQEvaluator.java:347) ~[identityiq.jar:8.3 Build f4b330b4da3-20220427-175259]
        ... 89 more
		

[kevinwoodbury]

We had the same issue, I too noticed a significant uptick in this. I really wish the Sailpoint API would be able to catch/handle these errors internally. I’ve found a few patterns that seem to cause this to happen.

1. Calls to context.decache(), especially while iterating over a list of objects.
2. Passing sailpoint.object.* between rules / functions. I’ve found that in some cases I’ve had to pass the ID of the object between rules and have to re-load the object with a context.getObjectById().

[drosenbauer (Devin Rosenbauer)]

More info…

I’ve found that they’re doing something new in the Provisioner that detaches the Identity object. I believe it’s related to locking. It helps to reload (just context.getObject, as you say) any impacted Identity and Link objects if you need to do more after invoking Provisioner.execute() against them.

It can help to invoke load() on an object before passing it around, but only if you don’t call decache() at some point. This will try to access a bunch of the lazily loaded lists within various objects, but I’ve found that it doesn’t necessarily get them all.

You can also try QueryOptions.setCloneResults(true).

If you want to have a totally detached version of the object, here’s a Java method I’ve written to ensure that the object is completely detached from any persistence. It just serializes the object to XML and then deserializes it back to an object.

	public static <T extends SailPointObject> T detach(SailPointContext context, T o) throws GeneralException {
		T retVal = (T) SailPointObject.parseXml(context, o.toXml());
		context.decache(o);
		return retVal;
	}

[mravisundar (Ravi Sundar)]

Hi,
After we upgrade into 8.2P1 we have also faced the same issue. As per SailPoint team suggestion

1. Don’t implement the context.decache(identityObj)
2. Don’t pass any identity object reference into WorkFlow.

[Ann_Fonseca_J (Ann Fonseca)]

Some links to additional info

• not passing objects in workflow steps: https://community.sailpoint.com/t5/Technical-White-Papers/BSDG-16-Workflow-Step-Bean-Shell-Best-Practices/ta-p/73229
• lazy init issues with 8.0 and after (specifically addressing commit during iterating) https://community.sailpoint.com/t5/IdentityIQ-Wiki/IdentityIQ-8-0-and-commitTransaction-While-Using-an-Iterator/ta-p/143225

[drosenbauer (Devin Rosenbauer)]

I had a customer open a ticket with support for this issue. We are seeing this fairly widespread across multiple IIQ installations. Reloading objects more often than necessary has fixed most of the issues we could address in code (e.g., Identity.links or Identity.roleAssignments).

However, we are most commonly getting errors lazy-loading Rule.referencedRules deep within Sailpoint code, which obviously we cannot change.

I’m debating whether we ought to just override the Rule.hbm.xml and set that field to NOT lazy-load. I suspect that would have a large performance hit, because Rule Libraries are huge, but I don’t know how Hibernate caching would handle it.

[ericlawson]

What was the solution that’s been applied to this problem? We are seeing this on things like Manage Accounts “Request Account” action, or common things in SailPoint not custom rules/workflows.