Tomcat Upgrade Issue - IdentityIQ Application Fails to Start

IdentityIQ (IIQ) IIQ Discussion and Questions
1

Hi everyone

I recently upgraded my Tomcat server from a manually installed version to an RPM-installed version. Here’s a brief overview of the process and the issues I’m encountering:

  1. Background:
  • Old Tomcat Version: Manually installed.
  • New Tomcat Version: Installed via RPM.
  • Application: SailPoint IdentityIQ.
  1. Upgrade Process:

  • Backup: I backed up the conf, logs, and webapps directories from the old Tomcat installation.

  • New Installation: Installed the new Tomcat version using RPM.

  • Restoration: Copied the webapps and conf directories from the old Tomcat installation to the new Tomcat directory.

  • Issue:

  • After starting the new Tomcat service, the SailPoint IdentityIQ application fails to start.

  • Logs:

  • I’ve reviewed the Tomcat logs and observed the following errors and warnings:

    • The identityiq context failed to start.
    • Warnings about potential memory leaks related to JDBC drivers and threads.
    • Errors indicating that some threads and resources were not cleaned up properly
[root@siiq-db-dev tomcat]# tail -n 50 /usr/share/tomcat/logs/catalina.2024-08-20.log                                                                                       
20-Aug-2024 23:50:57.508 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.87
20-Aug-2024 23:50:57.511 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Mar 11 2024 10:12:34 UTC
20-Aug-2024 23:50:57.511 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.87.0
20-Aug-2024 23:50:57.511 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
20-Aug-2024 23:50:57.511 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.18.0-553.8.1.el8_10.x86_64
20-Aug-2024 23:50:57.511 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
20-Aug-2024 23:50:57.512 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-17-openjdk-17.0.11.0.9-2.el8.x86_64
20-Aug-2024 23:50:57.512 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           17.0.11+9-LTS
20-Aug-2024 23:50:57.512 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Red Hat, Inc.
20-Aug-2024 23:50:57.512 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /usr/share/tomcat
20-Aug-2024 23:50:57.512 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/share/tomcat
20-Aug-2024 23:50:57.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/share/tomcat
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/share/tomcat
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.endorsed.dirs=
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/var/cache/tomcat/temp
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties
20-Aug-2024 23:50:57.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
20-Aug-2024 23:50:57.525 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
20-Aug-2024 23:50:57.751 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
20-Aug-2024 23:50:57.769 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-8443"]
20-Aug-2024 23:50:58.048 INFO [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-jsse-nio-8443], TLS virtual host [_default_], certificate type [RSA] configured from keystore [/etc/tomcat/keystore] using alias [tomcat] with trust store [null]
20-Aug-2024 23:50:58.050 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [704] milliseconds
20-Aug-2024 23:50:58.080 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
20-Aug-2024 23:50:58.080 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
20-Aug-2024 23:50:58.086 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/docs]
20-Aug-2024 23:50:58.474 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:50:58.496 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/docs] has finished in [410] ms
20-Aug-2024 23:50:58.496 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/examples]
20-Aug-2024 23:50:58.760 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:50:58.775 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/examples] has finished in [278] ms
20-Aug-2024 23:50:58.775 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/host-manager]
20-Aug-2024 23:50:58.911 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:50:58.914 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/host-manager] has finished in [139] ms
20-Aug-2024 23:50:58.914 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/identityiq]
[root@siiq-db-dev tomcat]# tail -n 50 /usr/share/tomcat/logs/catalina.2024-08-20.log 
20-Aug-2024 23:50:58.086 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/docs]
20-Aug-2024 23:50:58.474 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:50:58.496 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/docs] has finished in [410] ms
20-Aug-2024 23:50:58.496 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/examples]
20-Aug-2024 23:50:58.760 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:50:58.775 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/examples] has finished in [278] ms
20-Aug-2024 23:50:58.775 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/host-manager]
20-Aug-2024 23:50:58.911 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:50:58.914 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/host-manager] has finished in [139] ms
20-Aug-2024 23:50:58.914 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/identityiq]
20-Aug-2024 23:51:08.239 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
20-Aug-2024 23:51:08.240 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/identityiq] startup failed due to previous errors
20-Aug-2024 23:51:08.288 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [identityiq] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
20-Aug-2024 23:51:08.289 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [identityiq] appears to have started a thread named [Log4j2-TF-2-Scheduled-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
 [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)
 [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
 [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 [email protected]/java.lang.Thread.run(Thread.java:840)
20-Aug-2024 23:51:08.289 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [identityiq] appears to have started a thread named [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/java.lang.Object.wait(Native Method)
 [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
 com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:91)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 [email protected]/java.lang.Thread.run(Thread.java:840)
20-Aug-2024 23:51:08.290 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a03161b]) and a value of type [sailpoint.api.Meter.MeterSet] (value [sailpoint.api.Meter$MeterSet@1b44d3aa]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Aug-2024 23:51:08.298 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/identityiq] has finished in [9,384] ms
20-Aug-2024 23:51:08.298 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/manager]
20-Aug-2024 23:51:08.449 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:51:08.451 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/manager] has finished in [153] ms
20-Aug-2024 23:51:08.451 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/ROOT]
20-Aug-2024 23:51:08.577 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Aug-2024 23:51:08.578 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/ROOT] has finished in [127] ms
20-Aug-2024 23:51:08.581 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
20-Aug-2024 23:51:08.590 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-8443"]
20-Aug-2024 23:51:08.592 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [10542] milliseconds
20-Aug-2024 23:51:11.892 INFO [mysql-cj-abandoned-connection-cleanup] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load []. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1349)
                at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:981)
                at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:123)
                at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:90)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
                at java.base/java.lang.Thread.run(Thread.java:840)

Any help or insights would be greatly appreciated. Thank you!

2

Hi @Moben,

This problem can be for some reasons, make this check:

  • Owner of identityiq folder and subfolders (must be the account you launch tomcat with)
  • Owner and permission of tomcat folder
  • If the name of tomcat folder is changed, you must update the iiq configuration files
  • Check how tomcat process you have with ps aux | grep -i tomcat for example. Stop all process and start tomcat again

PS Before new installation, do you stop the old tomcat instance? Also, if you change the name of folder you need to change the /etc/systemd/system/tomcat.service file

2 Likes
3

Hi Moslem Benmahmoud,
Will you please tell me the IIQ Version?

4

We have been facing the same issue as well. Our client has performed similar steps to upgrade their tomcat. Have you got any resolution yet?

5

Hi @gvscdeep1,

upgrading tomcat is relatively easy and is uncorrelated from IIQ.
The steps are those:

  1. Backup of tomcat folder
  2. Stop tomcat service
  3. Remove\move the older version folder or unistall it
  4. Install the new version, better with the same name
  5. Copy conf & webapps folders from older to newer
  6. Start tomcat service

Also, you must copy other file like setenv into bin folder if have any particular configuration.

In every case, if you install with .exe in windows or extract it from zip file, use the same name for the service and for the folder. With this you dont need to change the configuration on IIQ.

In linux, you can create a SymLink and change the pointer of it to the newer folder and you dont need to move/delete the older.

2 Likes
6

Hi @Moben ,
Always prefer to copy or merge only those files that are modified from old conf folder to the new tomcat version conf folder.

For webapps as well just push the identityiq war or any modified file and remove unrequired apps from webapps.

Check if you have any modified files in bin folder eg setEnv or any Jars that are missing in tomcats lib folder.

Check if you JDBC driver is compatible with the new version.

Thanks

1 Like
7

Thank you all for your suggestions! The issue was caused by Tomcat being unable to make network connections via the DB driver, resulting in the error: com.mysql.jdbc.CommunicationsException: The driver was unable to create a connection due to an inability to establish the client portion of a socket.

To fix it, I enabled Tomcat to connect to the database by adjusting SELinux settings. I first checked the relevant SELinux booleans using getsebool -a | grep -i tomcat and then ran setsebool -P tomcat_can_network_connect_db on to allow network connections.

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.