Share all details related to your problem, including any error messages you may have received.
We have a number of API-friendly workflows we’ve made available in our IIQ instance to handle password resets, password changes, and application unlocks for other groups (usually our IT Support team, though our Intranet now uses one workflow to handle password changes).
We’ve noticed recently that around 1 out of every 10 API calls will take an extra 2-3 minutes during the Process Plan step of provisioning. It’s not consistent – the other 9 times out of every 10, the workflows will take between 7 and 15 seconds to complete – but it does happen a lot, and it only seems to happen on Process Plan steps. We also don’t see this during our normal Joiner/Mover/Leaver processing (all of which run with Perform Maintenance; the API Workflows are all on demand instead). It’s not consistent with any particular users or approvers, and there aren’t any work items that pop up. IIQ seems to see the call, and then wait somewhere between 30s and 3 minutes to complete the Process Plan provisioning phase.
We see this the most with the compilePasswordInterceptProject action from Sailpoint.Workflow.IdentityLibrary. I think it might be an issue with Approver lookup, as I’ve seen these tasks sit on “waiting for Approvers” at least once. Is there a way to tell that particular action to skip approval in certain circumstances? Or an approval setting that might be causing intermittent delays?
it could depends by many factors, we should see when it happens.
A little list of things for make an anilisys:
User logged number
server where is running the process
server usage in the moment
DB usage in the moment
running processes in the moment
are present locked object?
errors/warning into logs?
x PWI, look the WF, where setting the approve and delete it
Its not easy understand why a process on 10 need more time, but what you have to do is find an common element(if is a “quality of data” problem) or understand what the server is doing. The conclusion: study/active logs, monitoring, find eventually corrupted/locked data
We’ve already verified that it’s not specific to any of our servers, or associated with server utilization. We significantly increased the RAM, CPU, and threading available on these servers after we noticed the issue, but doing so didn’t seem to have any effect. It also doesn’t seem to happen consistently when other workflows or processes are running, and our DBAs have reported no obvious process spikes that could indicate a DB-side issue.
We aren’t setting approvers in the workflows that are being called. If there’s an approver step inside compilePasswordInterceptProject, that could be the issue, but compilePasswordInterceptProject is part of IIQ’s default IdentityLibrary. I’m not even sure if we can change that.
And yes, we have checked against our logs, and there aren’t any errors or warnings we can associate with the issue.
I’m not sure what you’re referring to with “user logged number”, however.
It could be that these workflow Are set to be run in background . When workflow is backgrounded those will wake only when “Perform Maintenance” job runs , generally these are scheduled to run every 2-5 mins .
You could try increasing the number of DB connections max total in iiq.properties file. 250 is ideal for medium installations and 350 for large installations
Abhishek:
We’re set to dataSource.maxTotal=300 at the moment… but our minIdle is only 5. Do you think increasing the number of idle, ready connections might make a difference?
Vishal:
They’re all set to Foreground Provisioning.
Hmm. We’re not connecting to the DB, but there are very specifically using two separate .getLinks() calls when checking for app accounts to update.
We aren’t currently using context.decache(); I thought we only needed that when updating an object, rather than just when building the list. I’ll add that in there and we’ll see what happens.
Also, is better if you use the search instead of the others, I remember of having read an topic where it was written search use less resources.
I found this article I will put here
Context.decache() helped, but I don’t believe it’s fully resolved the issue. I’m seeing fewer calls over 30s, but there are still a few of them.
The actual link check we’re doing is this:
List targetApplicationsRACF = new ArrayList();
List targetApplicationsLDAP = new ArrayList();
Identity identityLinkSearch = context.getObjectByName(Identity.class, identityName);
List links = identityLinkSearch.getLinks();
if((links != null) && (!links.isEmpty()))
{
Iterator iterator = links.iterator();
while(iterator.hasNext())
{
Link link = (Link)iterator.next();
Application app = link.getApplication();
if(app.getName().contains("DDS_RACF"))
{
targetApplicationsRACF.add(app.getName());
}
if(app.getName().contains("DDS_LDAP"))
{
if(app.getName().contains("DDS_LDAP_PROD"))
{
if(!checkLink(context,app.getName(),link.getNativeIdentity())){
throw new GeneralException("LDAP Prod is currently unavailable.");
}
}
targetApplicationsLDAP.add(app.getName());
}
}
wfcontext.setVariable("targetApplicationsRACF", targetApplicationsRACF);
wfcontext.setVariable("targetApplicationsLDAP", targetApplicationsLDAP);
I don’t think the two searches are the problem, however. Based on our logs, the slowdown occurs after both lists are calculated, around 3 steps later in the workflow. There is an Identity lookup and update before the slowdown (see below)