Tomcat automatically shut down for SailPoint 8.4 after the upgrade

IdentityIQ (IIQ) IIQ Discussion and Questions
,
1

Which IIQ version are you inquiring about?

SailPoint 8.4

Please share any images or screenshots, if relevant.

Please share any other relevant files that may be required (for example, logs).

Share all details about your problem, including any error messages you may have received.

Hi,

We have recently upgraded sailpoint from version 8.3 to 8.4. since we have upgraded 2 servers to new sailpoint. In one of the server tomcat abruptly shutsdown repeatedly. while in the other server it’s running fine. Can you please help on this. PFB the catalina logs
15-May-2025 08:25:15.556 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5b96933]) and a value of type [sailpoint.api.Meter.MeterSet] (value [sailpoint.api.Meter$MeterSet@19f71214]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.557 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5b96933]) and a value of type [sailpoint.api.Meter.MeterSet] (value [sailpoint.api.Meter$MeterSet@7f049f30]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.557 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@313a9112]) and a value of type [com.sun.faces.context.FacesContextImpl] (value [com.sun.faces.context.FacesContextImpl@107c5293]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.557 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [javax.faces.context.FacesContext$1] (value [javax.faces.context.FacesContext$1@239a4503]) and a value of type [org.primefaces.context.PrimeFacesContext] (value [org.primefaces.context.PrimeFacesContext@613fafc2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.557 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@313a9112]) and a value of type [com.sun.faces.context.FacesContextImpl] (value [com.sun.faces.context.FacesContextImpl@3a9941ca]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.558 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [javax.faces.context.FacesContext$1] (value [javax.faces.context.FacesContext$1@239a4503]) and a value of type [org.primefaces.context.PrimeFacesContext] (value [org.primefaces.context.PrimeFacesContext@5f4ef58a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.558 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@313a9112]) and a value of type [com.sun.faces.context.FacesContextImpl] (value [com.sun.faces.context.FacesContextImpl@2dd4dc3a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.558 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [javax.faces.context.FacesContext$1] (value [javax.faces.context.FacesContext$1@239a4503]) and a value of type [org.primefaces.context.PrimeFacesContext] (value [org.primefaces.context.PrimeFacesContext@310927fd]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.558 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@313a9112]) and a value of type [com.sun.faces.context.FacesContextImpl] (value [com.sun.faces.context.FacesContextImpl@416eb707]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.559 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [javax.faces.context.FacesContext$1] (value [javax.faces.context.FacesContext$1@239a4503]) and a value of type [org.primefaces.context.PrimeFacesContext] (value [org.primefaces.context.PrimeFacesContext@69a4f805]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.559 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@313a9112]) and a value of type [com.sun.faces.context.FacesContextImpl] (value [com.sun.faces.context.FacesContextImpl@621acc07]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.559 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [javax.faces.context.FacesContext$1] (value [javax.faces.context.FacesContext$1@239a4503]) and a value of type [org.primefaces.context.PrimeFacesContext] (value [org.primefaces.context.PrimeFacesContext@1ad42976]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.559 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@313a9112]) and a value of type [com.sun.faces.context.FacesContextImpl] (value [com.sun.faces.context.FacesContextImpl@2e3673a4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.559 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [javax.faces.context.FacesContext$1] (value [javax.faces.context.FacesContext$1@239a4503]) and a value of type [org.primefaces.context.PrimeFacesContext] (value [org.primefaces.context.PrimeFacesContext@601cc2b7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.560 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@54524047]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@6f764c22]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.560 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@54524047]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@7240a93b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-May-2025 08:25:15.560 SEVERE [Thread-172] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@54524047]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@74cfd4be]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Regards
Amit

2

@ayadav_12 are u able to connect iiq console from that box ? let me know.

3

Hi @ayadav_12,

stop the tomcat service, execute a killall command on java process and start(not restart) the tomcat process. Also, before of start of tomcat, delete the file that contains the pid of tomcat, usually on /var/run/tomcat.pid.

4

Compare Tomcat Configurations of 2 app servers:
Carefully compare the server.xml, context.xml, web.xml and any other relevant Tomcat configuration files between the working server and the problematic server.
Look for any discrepancies in: JVM arguments (especially memory settings like -Xms, -Xmx, -XX:MaxPermSize or -XX:MetaspaceSize depending on your Java version).

Compare Java Versions and Environment: Ensure that both servers are running the exact same Java version and operating system with the same patch levels.

see if the heap or other memory regions are filling up before the shutdown.

if not on latest patch 8.4p2 try to apply patch.

5

thanks!!

Also, i need help to resolve the below error which is comming during account aggregation:

Exception during aggregation. Reason: java.lang.RuntimeException: sailpoint.connector.ConnectorException: BeanShell script error: bsh.EvalError: Sourced file: inline evaluation of: import java.util.ArrayList; import java.util.Map; import java.util.HashMap; . . . '' : Typed variable declaration : Attempt to resolve method: extendedAttrServiceAccountEnabled() on undefined variable or class name: ObjectConfigAttributesRuleLibrary : at Line: 59 : in file: inline evaluation of: import java.util.ArrayList; import java.util.Map; import java.util.HashMap; . . . ‘’ : ObjectConfigAttributesRuleLibrary .extendedAttrServiceAccountEnabled ( context ) BSF info: Rule-Framework-RapidAppOnBoarding-Customization-Rule at line: 0 column: columnNo

Thanks
Amit

6

Hi @ayadav_12

  • Exception during aggregation.: The problem occurs when SailPoint IdentityIQ is trying to synchronize data from an application (likely Active Directory).
  • BeanShell script error: bsh.EvalError: ... Attempt to resolve method: extendedAttrServiceAccountEnabled() on undefined variable or class name: ObjectConfigAttributesRuleLibrary: This is the core issue. A BeanShell rule (which is what SailPoint uses for its dynamic logic) is trying to call a Java method named extendedAttrServiceAccountEnabled() on a class called ObjectConfigAttributesRuleLibrary. However, the BeanShell interpreter cannot find this class or method.
  • BSF info: Rule-Framework-RapidAppOnBoarding-Customization-Rule at line: 0 column: columnNo: This tells you the specific rule that is failing: Rule-Framework-RapidAppOnBoarding-Customization-Rule. The error occurs within this rule.

Why this happens (Most Common Reasons):

  1. Missing or Misplaced Custom JAR File: The ObjectConfigAttributesRuleLibrary is almost certainly a custom Java class developed for your SailPoint implementation. Its compiled form (a .jar file) needs to be placed in a specific location within your IdentityIQ deployment for the system to load it. If it’s missing, corrupted, or in the wrong place, you get this error.
  2. Application Server (Tomcat/JBoss) Restart Not Performed: Even if you put the JAR in the correct location, the application server needs to be restarted for it to pick up new or updated JAR files in its classpath. This is a very common oversight.
  3. Typo or Signature Mismatch: Less common if it worked before, but there might be a typo in the class name or method name, or the method’s arguments in the JAR don’t match how the rule is calling it (e.g., (context) argument).

Action Plan (Reiterated and Prioritized):

  1. Stop Aggregation: Cancel or stop any ongoing aggregation tasks that are failing to prevent continuous errors.
  2. Identify the JAR File:
  • Consult your development team/implementers: Ask them to provide the .jar file that contains the ObjectConfigAttributesRuleLibrary class. This is crucial. It might be a general customizations JAR (e.g., mycompany-custom-rules.jar, app-onboarding.jar).
  • Verify Class Presence (Advanced): If you have the JAR, you can use a tool like jar -tf yourjarfile.jar (from a command prompt where Java is installed) to list its contents and confirm ObjectConfigAttributesRuleLibrary.class is inside.
  1. Deploy the JAR to the Correct Location:
  • Navigate to your SailPoint IdentityIQ installation directory on your server(s).
  • The target directory for custom JARs is usually:
    • [IIQ_INSTALL_DIR]\webapps\identityiq\WEB-INF\lib\
  • Copy the JAR file you identified in Step 2 into this lib directory.
  • Important: If there’s an older version of this JAR already present, replace it with the new one. Ensure you have proper permissions to write to this directory.
  1. RESTART YOUR APPLICATION SERVER (CRITICAL!):
  • This is the most common reason for this error after deploying a new JAR.
  • Stop your Tomcat (or JBoss/WebSphere/WebLogic) server completely.
  • Wait a few moments.
  • Start your Tomcat server.
  • Ensure IdentityIQ comes up fully and is accessible.
  1. Re-run the Aggregation Task:
  • Once IdentityIQ is stable, go back into the UI and run the aggregation task for the Active Directory application again.

If the error persists after these steps:

  • Check Server Logs (beyond IIQ UI): Look at your Tomcat catalina.out (or equivalent for your app server) log files. They might provide more detailed information about JAR loading issues during startup.

Focus primarily on the JAR deployment and server restart. That’s where this kind of bsh.EvalError originating from an “undefined class” almost always lies.

7

Hi @ayadav_12

We are curious to know the status of the issue and would appreciate an update on your findings and the progress made, especially if the aforementioned steps assisted in its resolution.

8

Check if the class “ObjectConfigAttributesRuleLibrary” exists in any of the JAR files.

Additionally, if you have recently performed an upgrade, verify whether any JAR files have been discontinued or if there are any conflicts.