Same request executes multiple times on multiple VMs

IdentityIQ (IIQ) IIQ Discussion and Questions
, ,
1

Which IIQ version are you inquiring about?

8.4 p2

Context

We use workflow “Scheduled Assignment” to remove access, which is IT role, from Identity at defined Sun Set Date and Time.

Internally this is handled by request scheduled to execute mentioned workflow at Sun Set Date and Time.

We enabled saving Task Results by removing resultExpiration=“-1” XML attribute from Workflow element. So we see task results.

Issue

We see request sometimes (once or twice a day) executed multiple times on multiple VMs. We have 4 VMs executing requests.

Most of the errors happen due to 2 executions, however - sometimes we see 3 executions at the same time.

Executions happen very close in time (within 2 seconds) of each other.

On less loaded Staging environment this issue does not show.

Problem is that earliest execution does remove role, while everyone after fails because role is not on the Identity.

“Scheduled Assignment” workflow has number of steps between Compile Project and Provision.

We think it is concurrency issue.

Questions

Did anyone encountered similar situation?

Is there some configuration, except using only one VM for request execution, to help with this situation?

Thank you

2

Update is that Staging environment, with 3 WMs also executes same request on each VM. This was proved by enabling logs:

logger.sailpointRequestWorkflowRequestExecutor.name=sailpoint.request.WorkflowRequestExecutor logger.sailpointRequestWorkflowRequestExecutor.level=all logger.sailpointRequestRequestHandler.name=sailpoint.request.RequestHandler logger.sailpointRequestRequestHandler.level=debug logger.sailpointRequestRequestProcessor.name=sailpoint.request.RequestProcessor logger.sailpointRequestRequestProcessor.level=debug logger.sailpointRequestRequestService.name=sailpoint.request.RequestService logger.sailpointRequestRequestService.level=debug

3

Hi @opalyvoda what are you settings in iiq.properties for:

  • scheduler.quartzProperties.org.quartz.scheduler.instanceId
  • scheduler.quartzProperties.org.quartz.jobStore.isClustered

I believe the default is to not do any locking in the database, which can result in jobs being launched on more than one application server.

4

Hello Paul,

Thank you for response.

We do not have both: scheduler.quartzProperties.org.quartz.scheduler.instanceId and scheduler.quartzProperties.org.quartz.jobStore.isClustered in iiq.properties.

I looked up Google AI for
"scheduler.quartzProperties" "quartz.jobStore.isClustered"
and it is a very good candidate not only for Request but also for anything quartz manages in IdentityIQ.

Will test this,
Regards

5

Hi Oleh,

We are using these settings with good results:

  • scheduler.quartzProperties.org.quartz.scheduler.instanceId=AUTO
  • scheduler.quartzProperties.org.quartz.jobStore.isClustered=true

I’m puzzled why they are not the default settings.

6

More research showed that RequestProcessor class method
private void processRequest(SailPointContext context, ThreadPool pool, String id)
which looks like is the one, uses call
Request request = (Request)ObjectUtil.transactionLockById(context, Request.class, id);
Transaction lock seams to work only inside Hibernate Session, in other words - lock works within one VM.
For lock to mutually exclude between multiple VMs - it has to be persistent lock. This way it would go to DB and visible across VMs.

In addition, as far as I understand, Request processing is not using Quartz.

7

Hi Oleh, I believe transaction locks are implemented as database locks, so they should work between application servers.

Possibly there is something wrong with your hibernate settings?

8

@opalyvoda Paul rightly said transactions lock should lock the object across all threads and servers and if they try to access, they simply need to wait. While you fix the concurrency issue, as a stopgap solution you might want to add a transition script in your “Scheduled Assignment” to check if the user still has the access or not, if not, then you transition to end, otherwise move forward with the next step.

9

Hello Neelmadhav,

I agree with transaction going to DB.

While analyzing logs from 3 VM on millisecond level I noticed requests being executed sequentially.

I tried to simulate situation in our Staging today and this time there were no errors.

In production I see duplicated and errored out executions, not many. Starting from Feb 1 2026 until now failure rate is 0.963% or 0.00963 requests failing out of all executed.

I was thinking about Hibernate cache. It might be that cache is populated when requests start and each VM see role assigned to identity from cache.
I’ll try to drop cache in Start step and see if this helps.

Ideally when Plan gets compiled into Project IdentityIQ notices that role to be removed is not assigned anymore and does not include this role into project. Is this correct?

I’m not actively working on this issue, so will respond with some delays.

Thank you for following up,
I’ll try adding condition and skip significant part of workflow if identity does not have role to be removed.

10

Do you see a “Request” object created in the debug page for the deprovisioning of role? if so, you can see which VM it will run on? you will also see if multiple requests are created.