Does anybody have opinions on what to do with Tasks that are taking increasingly more time as we grow SailPoint?

Which IIQ version are you inquiring about?

Version 8.2

Please share any images or screenshots, if relevant.

Share all details related to your problem, including any error messages you may have received.

Hi all-

We have had this issue for awhile now and have never really been able to find a straight forward fix. Our Daily aggregation and refresh that we run overnight used to take ~4 hours to complete but is now nearly double that in the last 2-3 months to 7 hours. The refresh tasks within the sequential are at about a +150% difference from average.

We’ve increased RAM before

Currently in PROD:

The UI servers have 16GB RAM. This exceeds the recommendation by 50%.
The Task servers have 22GB RAM. This exceeds the recommendation by 37%.

Does anybody have any advice or input on why these are running longer and how we can remediate?

@colsmith The refresh tasks within the sequential are at about a +150% difference from average. - you mean when you run refresh task individually then it is faster?

My Thought, There should be Some Rule which we configured for Identity Mapping. It should be of Type IdentityAttribute Rule.

If it was replicating in lower env then remove all rule from Identity Mapping and run refresh and see if still you have issue or not ?

Thanks,
Pravin

1 Like

I would start with checking recommendations from this documents
https://community.sailpoint.com/docs/DOC-6655

and
https://community.sailpoint.com/docs/DOC-5428

If it doesn’t help you will need to identify process which is causing problems.

1 Like

We have several Identity Mappings with Application Rules. Do you think one of these rules is causing it to overflow?

I’ve narrowed the issue down to the “NM Identity Refresh - Refresh Attributes” task. It was taking 9 minutes on average to run completely and now it’s taking approximately 2 hours and counting.

Is there any reason you can think of for this huge growth?

@colsmith you have to narrow down on rules used in mapping. check all rules and remove which you see that might cause the problem first Or remove all rules from mapping and check the result.

Also check if any new changes recently did by someone.

Are these task partitioned ? How many thread and what is size of each partition?

Increasing the number of cores on sailpoint task servers and increasing the number of task servers helped us in moving task fast in our growing sailpoint env.

1 Like

@colsmith

Can you please share the task parameters of this particular task and also do you have any recent change of code or configuration in your target identity mappings

@vishal_kejriwal1

These tasks are partitioned. We have it set to 320 partitions.

Is there a good way to check the thread and size of each of these partitions?

May I ask how many Task Servers your envr currently runs?

@iamksatish

Absolutely.

So it’s the Identity Refresh - Refresh Attributes task and it does the following:

  • Do not reset the needing refresh marker after refresh
  • Refresh identity attributes
  • Refresh manager status

Partitioning enabled. Set at 320.

10 task servers and 2 UI servers

You can check the thread size in the Request Definition Identity Refresh Partition

Check the below recommendation related to RequestDefinition Object from link provided by @kjakubiak ( Partitioning Best Practices - Compass (sailpoint.com)) for configuring the threads based on the server CPUs you have

Note: A conservative number for maxThreads is one thread per CPU, but know that IdentityIQ has been tested with large numbers of partitions executing concurrently on servers with 4 CPUs. IdentityIQ is stable with up to 50 concurrent partitions on 4-CPU servers when running account aggregations. While the software is stable with this configuration, implementers should understand that overcommitting the CPUs of task servers like this (a 12.5 : 1 thread-to-CPU ratio in this example) usually does not result in optimal overall performance or task throughput. Implementers attempting to achieve optimal throughput should test their specific environment and hardware to find an optimal maxThreads value for each RequestDefinition object. A good number to start with is 2 times the CPU count for request servers. Thus, a typical medium-sized installation would have two task/request servers, each with 4 CPUs; 8 would be a reasonable starting point for each RequestDefinition object’s maxThreads value (ignoring each Server object’s maxRequestThreads setting because all task/request hosts have identical configuration). It is important to understand that each partition consumes a JDBC connection to IdentityIQ’s database for the duration of the task’s execution. Bear in mind that setting a reasonable task execution schedule to ensure proper thread management may require input from both technical and non-technical stakeholders. For instance, scheduling a large, partitioned Identity Refresh to run at the same time as multiple, partitioned aggregations every day may be ill-advised and require staggering of tasks schedules.

1 Like