Tomcat shutdown issue

IdentityIQ (IIQ) IIQ Discussion and Questions
1

Which IIQ version are you inquiring about?

Version 8.4

Share all details related to your problem, including any error messages you may have received.

Is anyone also encountering any tomcat shutdown issues using identityIQ 8.4. We previously were on 8.0 and have never seen this issues. Tomcat does not cleanly shutdown. We have IIQ 8.4, tomcat 9, jdk 11 and mysql 5.7

16-May-2024 10:42:43.611 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6c3c4a1a]) and a value of type [com.sun.faces.context.FacesContextImpl] (value [com.sun.faces.context.FacesContextImpl@3acea9d5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.612 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@41ca5c65]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.613 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@41ca5c65]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.614 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@41ca5c65]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.614 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@47b40c49]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.615 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@87b856b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.617 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@87b856b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.618 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@87b856b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.619 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@87b856b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.620 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [identityiq] created a ThreadLocal with key of type [connector.common.logging.LogContext$1] (value [connector.common.logging.LogContext$1@71dd3123]) and a value of type [connector.common.logging.LogContext] (value [connector.common.logging.LogContext@87b856b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
16-May-2024 10:42:43.635 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler [“http-nio-18080”]
16-May-2024 10:42:43.647 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler [“http-nio-18080”]
16-May-2024 10:42:48.585 INFO [mysql-cj-abandoned-connection-cleanup] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load . The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load . The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1052)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:123)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:90)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

2

Did you check if you have enough memory or if any code is causing the memory leak?

it could be alot of different things , even GC behavior.

3

Yes we have more than required memory. Servers never exceed 40% memory and 20%cpu utilization in our normal daily functioning. Also we have exact same code on parallel 8.0 setup where shutdown is clean.

4

@abhijeetsalvi

Check below thread

Solved: Task server CPU utilization is very high despite no tasks that are running - Compass (sailpoint.com)

5

As I said above, we dont have a high CPU or high memory utilization issue. Our utilization is very normal. Just that when we attempt to shutdown tomcat, it wont shut cleanly.

6

Are you on Linux? Once you’ve stopped Tomcat, are you able to see other java processes running?

After a ‘forced’ stop (“sudo kill -9 ”) and then a start, provide the following (if the severe is still there):

Maybe the following commands can provide more info:

list if any caching is there

find /work/ -type d

list all java processes

ps -ef|grep java

7

Yes we are on linux and can see the java process running even after tomcat shutdown. We have to kill the process manually using kill and then restart, so tomcat can restart properly, otherwise we end up getting multiple java processes. One of our developers was able to replicate this issue on windows too where tomcat 9 doesnt shutdown with IIQ 8.4. so I think it is not about platform.

We do see caching but isnt it normal for things to be cached.

8

Can you try to delete the application cache folder under /tomcat/work/Catalina/localhost and see if it works?!

9

Is this the “second” or more server in a cluster?

This looks a lot like the issues we have had with multiple servers.

Please advise I can help through this.

10

Hi @abhijeetsalvi,

When you trigger tomcat stop, iiq stop command will be issued. If there are any tasks running on the task servers, process will go into wait state and when kill these process ids then it will go into remove state and eventually tomcat service will be stopped.

systemctl stop apache-tomcat.service

systemctl status apache-tomcat.service

Using the above status command you can check what state is it and you can act accordingly.

Hope it helps.

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.