This article discusses a recent deployment issue with SailPoint IdentityIQ where the application failed to start after deployment. The root cause was identified as the “READ_COMMITTED_SNAPSHOT” setting being turned off for both the “IdentityIQ” and “IdentityIQ Plugin” databases. This setting is crucial for MS SQL Server environments, as specified in the SailPoint create script. Once the setting was enabled for both databases, the servers started successfully and became accessible.
Diagnosis
After deploying updates to SailPoint IdentityIQ, the application failed to start. The Tomcat logs were stuck at the following line with no errors:
25-Nov-2024 08:29:16.267 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/<path>/webapps/identityiq]
No further progress was made, and users were unable to access the IdentityIQ interface. After investigating and reviewing thread dumps with SailPoint Support, we discovered that the issue was related to a database configuration problem — specifically, the “READ_COMMITTED_SNAPSHOT” setting was disabled for both the IdentityIQ and IdentityIQPlugin
Troubleshooting Steps:
Troubleshooting Steps:
- Initial Observation:
- The application was stuck during the startup phase with no additional error messages in the
catalina.outlog. - The issue persisted even after restarting the Tomcat server and reviewing the application logs.
- Engaging SailPoint Support:
- We opened a support case with SailPoint and provided them with thread dumps captured via the
kill -3 <PID>command. - SailPoint Support analyzed the thread dump and confirmed that the main thread was waiting for a database response, suggesting a potential database lock issue or improper isolation level.
- Identifying the Problem:
- SailPoint’s analysis pointed us towards the READ_COMMITTED_SNAPSHOT isolation level for our MS SQL Server databases.
- The relevant configuration setting, IS_READ_COMMITTED_SNAPSHOT_ON, was found to be
0(disabled) for both the IdentityIQ and IdentityIQ Plugin databases.
- Database Isolation Settings in MS SQL Server:
- According to the SailPoint create script for IdentityIQ, READ_COMMITTED_SNAPSHOT must be enabled for both the IdentityIQ and IdentityIQ Plugin databases. This feature is essential for proper operation of IdentityIQ in MS SQL Server environments as it helps avoid blocking and improves concurrency.
- The following SQL script is used to ensure this setting is enabled:
sql
Copy code
ALTER DATABASE identityiq SET READ_COMMITTED_SNAPSHOT ON;
ALTER DATABASE identityiq_plugin SET READ_COMMITTED_SNAPSHOT ON;
- Snapshot isolation (another related setting) was already enabled, so no changes were needed for that.
- Resolution:
- After running the above SQL commands to enable READ_COMMITTED_SNAPSHOT for both databases, we restarted the Tomcat
