Account Aggregation Completed Trigger - Not firing on error, showing success on warning

mcheek · January 11, 2024, 7:11pm

Note that I have already created a support case for this issue.

I have a process that utilizes the account-aggregation-completed event trigger. It creates a ticket in ServiceNow whenever there is an error or warning so we can address the issue. I noticed today there were failures and warnings with no tickets created, so I dug a little deeper.

I created a new trigger without any filtering so I could make sure to eliminate any issues with the filter

Error not firing trigger at all
I had a source with a failed account aggregation

I could see that reflected in both the /cc/api/event/list endpoint

{
    "id": 4654893,
    "type": "CLOUD_ACCOUNT_AGGREGATION",
    "status": "ERROR",
    "timestamp": "2024-01-11T17:06:51Z",
    "dateCreated": "2024-01-11T17:06:51Z",
    "details": {
        "name": "Cloud Account Aggregation - 770ae5688a544bbfa914b024ab657397",
        "description": null,
        "messages": {
            "Error": "com.sailpoint.mantisclient.exception.baserestclient.BaseRestClientConnectionFailedException: Unable to execute request to URI https://prd04-useast1-shelby.accessiq.sailpoint.com/sources/fd3a6d24b06242ffad1265c4c95cf6c7/native-change-detection-config"
        },
        "attributes": {
            "eventId": 4654893,
            "clusterCcgBuild": "937",
            "appId": "fd3a6d24b06242ffad1265c4c95cf6c7",
            "optimizedAggregation": "enabled"
        },
        "completed": 1704992930139,
        "id": "4ce8a43601654c03ba965241634d22d5",
        "launched": 1704992810760,
        "status": "Error"
    },
    "user": {
        "alias": "SYSTEM",
        "externalId": "2c9180846d170ded016d888a96eb160d"
    },
    "object": {
        "id": 1851555,
        "externalId": "fd3a6d24b06242ffad1265c4c95cf6c7",
        "type": "source",
        "displayName": "Azure Active Directory - Guests"
    }
}

As well as the audit log

{
    "org": "chk",
    "pod": "prd04-useast1",
    "created": "2024-01-11T17:08:50.243Z",
    "id": "4142e2ebd5464f56a042abcefe8201b1",
    "action": "SOURCE_ACCOUNT_AGGREGATION_FAILED",
    "type": "SOURCE_MANAGEMENT",
    "actor": {},
    "target": {
        "name": "Azure Active Directory - Guests [source]"
    },
    "stack": "qpoc",
    "trackingNumber": "44d7a2d7-fe80-4c6a-a6fb-de84597f8eee",
    "attributes": {
        "aggregationId": "4ce8a43601654c03ba965241634d22d5",
        "sourceId": "fd3a6d24b06242ffad1265c4c95cf6c7",
        "sourceName": "Azure Active Directory - Guests"
    },
    "objects": [
        "SOURCE",
        "ACCOUNT"
    ],
    "operation": "AGGREGATE",
    "status": "FAILED",
    "technicalName": "SOURCE_ACCOUNT_AGGREGATE_FAILED",
    "name": "Aggregate Source Account Failed",
    "synced": "2024-01-11T17:08:50.403Z",
    "_type": "event",
    "_version": "v7"
}

However, there were no trigger invocations fired on that trigger subscription for this aggregation failure

Warnings showing as successful

I had another source end in a warning because of an account deletion threshold

The warning status was reflected in the /cc/api/event/list endpoint

{
    "id": 4654940,
    "type": "CLOUD_ACCOUNT_AGGREGATION",
    "status": "WARNING",
    "timestamp": "2024-01-11T18:08:12Z",
    "dateCreated": "2024-01-11T18:08:12Z",
    "details": {
        "name": "Cloud Account Aggregation",
        "description": null,
        "messages": {
            "Warn": "Account deletion was skipped for Automic because the threshold of 36 was exceeded."
        },
        "attributes": {
            "eventId": 4654940,
            "total": "97",
            "clusterCcgBuild": "937",
            "optimized": "97",
            "appId": "2c91808481d42b010181d90cbcd9662c",
            "optimizedAggregation": "enabled"
        },
        "completed": 1704996495844,
        "id": "99b563fdea14423eb45a79d9ab830c0b",
        "launched": 1704996491280,
        "status": "Warning"
    },
    "user": {
        "alias": "SYSTEM",
        "externalId": "2c9180846d170ded016d888a96eb160d"
    },
    "object": {
        "id": 1847540,
        "externalId": "2c91808481d42b010181d90cbcd9662c",
        "type": "source",
        "displayName": "Automic"
    }
}

And there was a trigger subscription invocation for it as well. But curiously, the status of this shows ‘Success’, even though the event above shows warning, granted, it does have a warnings property included

{
  "_metadata": {
    "invocationId": "05dfb187-cfae-4bc9-b1d9-84b054b1ca17",
    "triggerId": "idn:account-aggregation-completed",
    "triggerType": "fireAndForget"
  },
  "completed": "2024-01-11T18:08:16.048615Z",
  "errors": [],
  "source": {
    "id": "2c91808481d42b010181d90cbcd9662c",
    "name": "Automic",
    "type": "SOURCE"
  },
  "started": "2024-01-11T18:08:11.280Z",
  "stats": {
    "added": 0,
    "changed": 0,
    "removed": 0,
    "scanned": 97,
    "unchanged": 97
  },
  "status": "Success",
  "warnings": [
    "Account deletion was skipped for Automic because the threshold of 36 was exceeded."
  ]
}

The documentation seems to suggest that such an aggregation should not have a Success status

So what gives? Anyone else experiencing this? Am I misunderstanding how this should work?

edmarks · January 11, 2024, 11:12pm

I’m assuming you’re using some kind of filter and branching logic for each of these types in the workflow?

One thought is removing this filtering logic and just send an email to yourself with the data it’s getting to ensure you’re getting the values you’re expecting based on the documentation.

mcheek · January 11, 2024, 11:26pm

I created a new trigger subscription without any filter to troubleshoot, and I’m getting the same results, as in the trigger isn’t firing at all on errors, and warnings are firing, but showing a status of success.

I’m also not triggering a workflow, I’m having it send to a webhook on webhook.site which is what the documentation recommends for testing

mcheek · January 12, 2024, 3:42pm

Ok, I was able to reproduce this in my sandbox in 1/2 scenarios

First, here is the trigger subscription (webhook url obfuscated)

{
        "triggerId": "idn:account-aggregation-completed",
        "httpConfig": {
            "httpDispatchMode": "SYNC",
            "httpAuthenticationType": "NO_AUTH",
            "url": "https://webhook.site/someRandomGUID",
            "basicAuthConfig": null,
            "bearerTokenAuthConfig": null
        },
        "triggerName": "Account Aggregation Completed",
        "description": "",
        "enabled": true,
        "id": "818a547b-8ccc-4b4a-beac-a38cc6ff735c",
        "type": "HTTP",
        "name": "Test Agg"
    }

Warnings showing as success

This is a JDBC source I was testing with, so I changed the accounts query adding a where clause that would result in 0 rows being returned, and thus a deletion of all the accounts. The threshold is currently set at 10%

{
    "SQL": "SELECT LOWER(REPLACE([UserName],'CHKENERGY\\','')) [UserName],[RoleValue] [Role] FROM [WellReadiness].[dbo].[UserSecurity] where UserName = 'definitelyWontMatch' order by UserName"
}

Ran the account aggregation, ended in a warning

However, the message sent to the webhook shows a status of Success

{
  "_metadata": {
    "invocationId": "87258fc7-73a3-48c7-a3c4-247d0a8e412d",
    "triggerId": "idn:aggregation-accounts-collected",
    "triggerType": "fireAndForget"
  },
  "completed": "2024-01-12T15:21:27.942933Z",
  "errors": [],
  "source": {
    "id": "2c91808581a6c2b50181b5c0112f2e1a",
    "name": "WellReadiness DB",
    "type": "SOURCE"
  },
  "started": "2024-01-12T15:21:24.412Z",
  "stats": {
    "added": 0,
    "changed": 0,
    "removed": 0,
    "scanned": 0,
    "unchanged": 0
  },
  "status": "Success",
  "warnings": [
    "Account deletion was skipped for WellReadiness DB because the threshold of 23 was exceeded."
  ]
}

Errors not firing

I changed the SQL in the account query once again, this time to have a syntax error

{
"SQL": "SELECT LOWER(REPLACE([UserName],'CHKENERGY\\','')) [UserName],[RoleValue] [Role] FROM [WellReadiness].[dbo].[UserSecurity] where UserName = 'definitelyWontMatch order by UserName"
}

Ran account aggregation, ended in error

This time a message was sent to the webhook with an error status

{
  "_metadata": {
    "invocationId": "7baa438c-5f6d-4ecb-9213-9ee33d4bdf7f",
    "triggerId": "idn:aggregation-accounts-collected",
    "triggerType": "fireAndForget"
  },
  "completed": "2024-01-12T15:29:28.174101Z",
  "errors": [
    "[ ConnectorException ] \n [ Error details ] The server encountered an unexpected error while contacting target system. Please check the logs. Unclosed quotation mark after the character string 'definitelyWontMatch order by UserName'."
  ],
  "source": {
    "id": "2c91808581a6c2b50181b5c0112f2e1a",
    "name": "WellReadiness DB",
    "type": "SOURCE"
  },
  "started": "2024-01-12T15:29:24.795Z",
  "stats": {
    "added": 0,
    "changed": 0,
    "removed": 0,
    "scanned": 0,
    "unchanged": 0
  },
  "status": "Error",
  "warnings": []
}

I noticed in my production tenant, the errors on aggregation appeared to be after the data was fetched from the source and the aggregation attempted to make whatever inserts/updates it needed to on the IdentityNow side

{
    "id": 4654893,
    "type": "CLOUD_ACCOUNT_AGGREGATION",
    "status": "ERROR",
    "timestamp": "2024-01-11T17:06:51Z",
    "dateCreated": "2024-01-11T17:06:51Z",
    "details": {
        "name": "Cloud Account Aggregation - 770ae5688a544bbfa914b024ab657397",
        "description": null,
        "messages": {
            "Error": "com.sailpoint.mantisclient.exception.baserestclient.BaseRestClientConnectionFailedException: Unable to execute request to URI https://prd04-useast1-shelby.accessiq.sailpoint.com/sources/fd3a6d24b06242ffad1265c4c95cf6c7/native-change-detection-config"
        },
        "attributes": {
            "eventId": 4654893,
            "clusterCcgBuild": "937",
            "appId": "fd3a6d24b06242ffad1265c4c95cf6c7",
            "optimizedAggregation": "enabled"
        },
        "completed": 1704992930139,
        "id": "4ce8a43601654c03ba965241634d22d5",
        "launched": 1704992810760,
        "status": "Error"
    },
    "user": {
        "alias": "SYSTEM",
        "externalId": "2c9180846d170ded016d888a96eb160d"
    },
    "object": {
        "id": 1851555,
        "externalId": "fd3a6d24b06242ffad1265c4c95cf6c7",
        "type": "source",
        "displayName": "Azure Active Directory - Guests"
    }
}

I was hoping someone from @developer_advocates might weigh in on two questions I have here

The documentation for account-aggregation-completed suggests that an aggregation that ends in a warning due to deletion threshold should have a status of ‘Termination’, however, I’m seeing those come over as ‘Success’
Aggregations that error out on the IdentityNow tenant side do not appear to trigger a message at all. This is problematic because I have an audit control around aggregation failures, and I don’t currently have an adequate explanation of why an aggregation failure would show up in the audit log, but not trigger an incident record to be created. Is this expected behavior?

colin_mckibben · January 12, 2024, 8:44pm

Thanks for bringing this to our attention. I’m going to have a hard time reproducing this myself, but support should be able to assist with reproduction and sending this to the right engineering team. Can you please open a support ticket?

mcheek · January 12, 2024, 8:54pm

I have, I was hoping to get some input on whether or not it was expected behavior and if anyone else had noticed it

system · March 12, 2024, 8:55pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Workflow - Account aggregation completed ISC Discussion and Questions workflows , identity-security-cloud	7	1028	July 19, 2023
Aggregation Failure Report Alert ISC Discussion and Questions aggregation , identity-security-cloud	10	1013	September 15, 2023
Entitlement Aggregation Failure ISC Discussion and Questions workflows , identity-security-cloud	2	255	December 31, 2023
Identity Aggregation Completed ISC Discussion and Questions	5	1352	July 19, 2023
Inquiry about Setting Up an Email Notification for Account and Permission Aggregation Failures ISC Discussion and Questions aggregation , identity-security-cloud , failure	9	56	November 26, 2024

Account Aggregation Completed Trigger - Not firing on error, showing success on warning

Related topics