Increased runtime of daily refresh task after the upgrade to 8.5

IdentityIQ (IIQ) IIQ Discussion and Questions
1

Which IIQ version are you inquiring about?

8.5p2

Please share any images or screenshots, if relevant.

We are running the following settings set as true in our daily refresh task.

<entry key="correlateEntitlements" value="true"/>
<entry key="correlateScope" value="true"/>
<entry key="enablePartitioning" value="true"/>
<entry key="partitions" value="256"/>
<entry key="processTriggers" value="true"/>
<entry key="promoteAttributes" value="true"/>
<entry key="promoteManagedAttributes" value="true"/>
<entry key="provision" value="true"/>
<entry key="refreshIdentityEntitlements" value="true"/>
<entry key="refreshManagerStatus" value="true"/>
<entry key="refreshRoleMetadata" value="true"/>
<entry key="synchronizeAttributes" value="true"/>

lease share any other relevant files that may be required (for example, logs).

No new errors shown in logs compared to 8.4p2.

Share all details about your problem, including any error messages you may have received.

Before the upgrade from 8.4p2 to 8.5p2, our refresh task that refreshes about 150k users took 4 hours to complete. After the upgrade, with no changes to the task, it took 25h to complete. We´ve also checked and added logging to any of triggers and rules that are running when refreshing a user and we can see no reason in our logs why the refresh of each user should take a lot longer.

Has anyone had a similar experience when upgrading to 8.5 or any other version? Or knows if something has changed with any of the refresh settings in the latest version?

2

Set refreshRoleMetadata to false and run the Identity Refresh task again. See if it improves the performance of the Identity Refresh task. What is the approximate count of roles you have in your environment?

3

Thanks for the suggestion, we will try that first thing tomorrow. We have little over 11k roles at the moment.

4

Hi @falin
The dramatic jump from 4 hours to ~25 hr is certainly caused by the over-inclusive task configuration combined with version-specific architectural changes in entitlement correlation, partitioning, and role metadata indexing.
Recommended fix for now

Split tasks into:
1.daily refresh
a)promoteAttributes=“true”
b)processTriggers=“true”

2.Weekly/Targeted maintance
a)correlateEntitlement=“false”
b)refreshIdentityEntitlements=“false”

Reduce partitions to rougly 2x to 4x to available task worker therds across your cluster.
verify db post upgrade index rebuilds, and db table statistics(ex:spt_identity,spt_entitlement, spt_link) stale excution plans are common in mysql/Oracle post upgradation to slowdown the bulk hibernate loads.

Please free to ask your queries.

5

Hi Felix, is there an improvement in the performance of the Identity Refresh Job after setting refreshRoleMetadata to false?

6

Hi Pavan,

Thanks for the response, we will test splitting up the task. Do you have any input on the remaining settings on the task? If they should be on the daily or weekly refresh?

Best regards,
Felix

7

Hi Vamsee,

We did not see an improvement when setting the refreshRoleMetaData to false, so most likely something else behind the scenes that is the root cause.

Best regards,
Felix

8

Hi @falin, One more suggestion: Can you reduce the number of partitions to 32 and check the performance of the identity refresh job?

9

@falin Could you please share what all options you have selected in refresh task?
You should also consider running delta refresh task and also create split task into multiple ones, in case you are doing everything in a single task.

10 
Hi,

These are all the options we have in the original task.
<entry key="TaskDefinition.runLengthAverage" value=""/>
<entry key="TaskDefinition.runLengthTotal" value=""/>
<entry key="TaskDefinition.runs" value=""/>
<entry key="TaskSchedule.host"/>
<entry key="checkHistory" value="false"/>
<entry key="checkPolicies" value="false"/>
<entry key="correlateEntitlements" value="true"/>
<entry key="correlateScope" value="true"/>
<entry key="deleteDormantGroups" value="false"/>
<entry key="disableManagerLookup" value="false"/>
<entry key="doManualActions" value="false"/>
<entry key="enableManualAccountSelection" value="false"/>
<entry key="enablePartitioning" value="false"/>
<entry key="excludeInactive" value="false"/>
<entry key="filter" value="name == 'testrefresh'"/>
<entry key="filterNeedsRefresh" value="false"/>
<entry key="forceWorkflow" value="false"/>
<entry key="includeWindowModified" value="false"/>
<entry key="keepInactiveViolations" value="false"/>
<entry key="markDormantScopes" value="false"/>
<entry key="maxExceptions" value="0"/>
<entry key="noAutoCreateScopes" value="false"/>
<entry key="noMaintenanceWindowRetry" value="false"/>
<entry key="noResetNeedsRefresh" value="false"/>
<entry key="noRoleDeprovisioning" value="false"/>
<entry key="processTriggers" value="true"/>
<entry key="promoteAttributes" value="true"/>
<entry key="promoteManagedAttributes" value="true"/>
<entry key="provision" value="true"/>
<entry key="refreshCertifications" value="false"/>
<entry key="refreshCompositeApplications" value="false"/>
<entry key="refreshGroups" value="false"/>
<entry key="refreshIdentityEntitlements" value="true"/>
<entry key="refreshManagerStatus" value="true"/>
<entry key="refreshRoleMetadata" value="true"/>
<entry key="refreshScorecard" value="false"/>
<entry key="refreshThreads" value="8"/>
<entry key="synchronizeAttributes" value="true"/>
<entry key="taskCompletionEmailNotify" value="Disabled"/>
<entry key="taskCompletionEmailRecipients"/>
<entry key="taskCompletionEmailTemplate"/>

During last Friday we split up the task to three smaller ones, two refreshes that run fast with:
1)

<entry key="processTriggers" value="true"/>

<entry key="promoteAttributes" value="true"/>


2) 
<entry key="correlateScope" value="true"/>

<entry key="refreshManagerStatus" value="true"/>

3) Running slow

<entry key="correlateEntitlements" value="true"/>

<entry key="enablePartitioning" value="true"/>

<entry key="provision" value="true"/>

<entry key="synchronizeAttributes" value="true"/>

Currently we have let IIQ set the number of partitions to 160, but I will test with a smaller set of partitions to see if that affects performance.
11

@falin Can you try splitting it like this:

  1. promoteAttributes, refreshManagerStatus & correlateScope
  2. processTriggers
  3. correlateEntitlements & provision
  4. synchronizeAttributes

You should set the partitions based on the number of servers you have. Please go to : RequestDefinition → Identity Refresh Partition. See how many parititons you have per server.

12

We are experiencing the same issue here. It appears that IIQSR-980, referenced in the 8.5p2 README file, introduced a problem with the CorrelationModel. The model never loads after startup.

SailPoint provided the IIQSR-995 and IIQSR-997 e-fixes, but they did not resolve the issue in our environment. The problem continues to occur.

13

We just upgraded and are experiencing same issue, our refresh task with refresh assigned… + provision assignments have more than doubled in execution time after upgrade. Just created a ticket to support, lets see what they say..

14

Hi Pèrsio!

Thanks for your reply.

What we’ve observed in our client’s environment is that the correlateEntitlement and provision parameters in the Identity Refresh task appear to be responsible for the significantly increased runtime following the upgrade to 8.5p2.

We have an ongoing support case with SailPoint, but they have not yet identified the root cause or suggested a fix.

15

Hello guys,

After comparing CorrelationModel.class 8.5p1 and 8.5p2 with AI assistance, we identified the architectural change.

In 8.5p1, CorrelationProfiles were eagerly loaded in the CorrelationRole constructor.
In 8.5p2, role details, selectors and profiles are loaded lazily through:

  • ensureCorrelationProfilesLoaded()
  • loadProfilesByRoleId()
  • warmBundleProfilesFromLoadedProfiles()
  • warmBundleProfileCollection()

There was blocked thread dumps collected in our environment consistently show execution inside ensureCorrelationProfilesLoaded().

All tasks using correlateEntitlements=true appear to wait on this new lazy-loading path.

We also asked to Engineering via Support Ticket to confirm if new CorrelationModel implementation depends on BundleProfileRelation data during profile warm-up and if a supported workaround exists to disable or bypass the lazy-loading mechanism and revert to the previous eager-loading behavior used in 8.5p1.

No answers so far.

18

Hi Persio! Thank you for the great summary. We have today been adviced by Sailpoint to apply the fixes IIQSR-995 and IIQSR-997 to fix the defects of the CorrelationModel. I saw that you have been adviced to to the same but still face significly slower run times of the refresh task when using correlateEntitlements flag.

Have you heard back from SP and do you have an open Support ticket with them currently?

Thanks in advance!

19

Hello Nils,

The only recommendation SailPoint has given us so far is to move correlateEntitlements to a separate step in the workflow, possibly as a subprocess. They also explained that the CorrelationModel now depends, at least in part, on the BundleProfileRelation data. It appears that the “new” CorrelationModel is now built on top of this service.

So far, in our testing, this approach has not improved or resolved the issue. We have several workflows that depend on correlateEntitlements, and they still hang during execution.

Another suggestion was to run a full Identity Refresh with correlateEntitlements enabled during off-hours on a daily basis. However, this is not practical in our environment due to the large number of bundles we have.

We are still trying to determine the best way to address this issue.

Regards,
Pérsio

20

Hi again,

Thank you for your swift response and detailed answer!

The suggestion from SailPoint to run a full Identity Refresh with only correlateEntitlements enabled during off-hours does not seem like a viable workaround for larger environments.

We have tested this in our client’s environment, and the task takes 18+ hours to complete, while also consuming significant application server resources during execution. Running this on a daily basis would therefore not be practical for us.

We will continue to update our support case with SailPoint and share any findings from our side. Please feel free to do the same if you make any progress in your investigation.

Thanks again for sharing your findings!

21

@persio_hartmann and if anyone else is following this thread - Sailpoint have now assigned this issue to Engineering Tracking Number (ETN) IIQETN-13206.

22

Hi @nilsgunnarson! Do you know the specifics on what this ETN aims for? Is there an e-fix associated with?

Best regards!