EPIC EMP Aggregation was failing Sporadically

Identity Security Cloud (ISC) ISC Community Knowledge Base
, ,
1

Problem

The EPIC EMP Connector was integrated with ISC and suddenly when all the other sources were scheduled for aggregation, EPIC EMP was sporadically getting Failed for 70% instances and was succeeded only 30% of runs which became a hug blocker issue for us. The exception that was getting thrown from EPIC EMP InterConnect APIs are as follows.

sailpoint.connector.InvalidResponseException: [ InvalidResponseException ] [ Possible suggestions ] Not working SOAP service… [ Error details ] Failed to process response…null

Diagnosis

During the diagnosis with SailPoint, we found that few of the Aggregation Setting Parameters and Virtual Appliances No. Of Processors and RAM/Memory was not as per SailPoint Recommendations

Solution

Part-1

  1. Login to SailPoint ISC as Administrator.
  2. Go to Admin → Connections → Sources
  3. Search for EPIC EMP connector specific source
  4. Open the EPIC EMP Source
  5. Go to Aggregation Settings
  6. Make changes to the following parameters as below.
Sr. # Parameter Name Parameter Value Comments
1 Page Size 1000
2 Aggregation Timeout 7200 This timeout will depend upon number of records which you have in EMP. Hence, calculate the average aggregation time from past 10 - 15 aggregations and then, provide a value. For us, it took 2 hours for nearly 50k records
3 Partition Aggregation Mode Disable
4 Retry Error List Read timed out
Connection reset
The input stream for an incoming message is null
5 Maximum Number of Retries 5
6 Wait Time for Retry 10000
  1. Save the changes
  2. Go to Advanced Settings
Sr. # Parameter Name Parameter Value
1 Maximum Number of Calls Permitted to Core Web Services 5000
2 HTTP Client Connection Timeout 60000
3 HTTP Client Socket Timeout 60000
  1. Save the changes.
  2. Validate the changes in EPIC EMP source in ISC and Test the Connections

Part-2

  1. Also, we saw that Virtual Appliances were 2 processors and 8 GB RAM. Hence, we increased it to 8 Processors and 16 GB RAM and restarted the VAs
  2. Post the above changes, aggregation was successful for nearly 10 runs in a row and issue was resolved.

Long Term Solution

  1. A separate VA Cluster with 2 VAs of 8 Processors and 16 GB RAM with 128 GB Storage has to be created and attached to EPIC EMP sources as these are very important and heavy sources for HealthCare
  2. Note that this new VA Cluster should not be attached to EPIC SER as EPIC SER fetches the aggregation data from File Extract
  3. Also, we need to make sure that Aggregation schedule has to be in such a way that it has to be done after authoritative source and EPIC EMP aggregation has to be secluded from other sources for sure.
1 Like