Workflow Background Steps Executing Multiple Times on Task Servers

IdentityIQ (IIQ) IIQ Discussion and Questions
1

We have two Task Servers and two UI servers. We have exposed APIs in SailPoint IIQ that are invoked by ServiceNow. When the API is called, a workflow is launched on one of the UI server. The first step runs immediately on the UI server, and then the workflow transitions to a background step:

<Step background="true" icon="Start" name="Start" posX="12" posY="13">
<Transition to="Log Input Arguments"/>
</Step>

Expected behavior:

  • Workflow starts on the UI server.
  • Once it transitions to background, execution continues on the task server.

However, in a recent occurrence:

  • The workflow started only once on the UI server.
  • But I am seeing three occurrences of the subsequent steps executing on task servers.

ServiceNow confirmed the API was called only once, and the workflow appears to have been initiated only once.

This issue is new and has never happened before in our environment.

Has anyone experienced similar behavior? What could cause background steps to execute multiple times on task servers for a single workflow instance?

Any insights would be appreciated.

2

It’s the “Perform maintenance” task that processes the background workflows. Could you have more than one instance of it running at a time?

3

Thank you, Paul for quick response. I found only one occurrence of PM task around that time.

4

@r_pragati, How many task results are generated when calling the SailPoint API? Is it one or three task results that are generated? You can check in the task result how many times it’s hitting SNOW. However, we have a similar implementation. I hope you can distribute the load across the UI servers and the Task server by enabling and disabling the system configuration. You can check the system configuration in the Administrator console, the UI server and the Task server both have enabled Request and Task flags, which will distribute the load across the servers.

Thanks,

PVR.

5

@r_pragati Have you enabled trace? Is it possible for you to share the workflow for review?

6

I see only one task result.

7

Sure, it’s hitting the workflow only once since the task result is once. May I know the reason why you use the background=" flag in the starting step? Is there any business test case you are handling? If not, that would be the culprit to run this step on other servers. If you are not handling any business case based on the background=“true” flag, I would suggest that you remove the background=“true” and try it. Workflow steps will not switch between the servers. you can track in Task results too

8

This workflow is invoked via API and we dont want to overload the UI servers, thats why we have background=“true”.
We are unable to reproduce the issue again.

9

Yes, workflow trace is enabled. I will share logs soon.

10

issue resolved, or does it still persist? You confirmed you were unable to reproduce the issue.

Thanks,

PVR.

11

This issue was one-off occurrence and did not occur again. Since the API and workflow have been working seamlessly for more than a year, I am trying to find the RCA.

1 Like
12

Possibly it is due to multiple perform maintenance trying to resume the workflow. Please also check if there is an overlap of PM tasks.

1 Like