Get List of Identities breaking loop in Workflows

[mosareini (Mohamad Sareini)]

Hello,

I am currently building a workflow that will create ServiceNow tickets for identities that are not getting role based access. We have a saved search that checks for users who are not receiving access through a job title role. My workflow is meant to get that list of identities, loop through them and create a servicenow ticket for each identity. The issue I am running into, is that the Get List of Identities step in the workflow, is returning too much data that I do not need in the response body. The query only returns 3 identities but the response body has every attribute, every entitlement, every role, etc. that is associated to the identities. I believe my loop is failing due to the excess data in the response body. This is what is happening when I test the workflow.

Here is my loop input

I know the loop operator has a limit of 100 items in the array that is passed to it, which is what I think is making this loop fail, although I could be wrong. I was wondering if this is the normal output for the Get List of Identities tile, and if so, how can I limit the response to only the fields I need, so that I can pass that to my loop? Any help would be much appreciated, thank you!

[margocbain (Margo Bain)]

To validate if it’s the size that is causing the issue and nothing else, try limiting your input to $.getListOfIdentities.identities[0:10] (as an example).
If you only need the id of the identities for your HTTP Request, try changing to $.getListOfIdentities.identities[*].id, this will simplify your input.
If you need to loop through a large data set that exceeds the loop input maximum, take a look at this article: Recursive Workflows in IdentityNow for asynchronous and synchronous recursion. It can be helpful if you need to execute on a large list of items with an unknown size, and there are great examples and code artifacts to help.

[colin_mckibben]

@mosareini

I was wondering if this is the normal output for the Get List of Identities tile, and if so, how can I limit the response to only the fields I need, so that I can pass that to my loop?

Hi Mohamad,

Yes, the Get List of Identities action will return all attributes of an identity, which can be an enormous amount of information depending on how many accounts and access your identities have. At the time of my reply, Workflows have the following limitation:

The maximum allowed size for a workflow definition is 400KB. The maximum allowed size for a workflow definition plus its input is 1.5MB

Additionally, loops have the following limit at the time of this reply:

The array you select can contain no more than 100 items. If an item in this list is larger than 512KB, that item will fail when the loop is executed.

Get List of Identities relies on the search API under the hood, which, by default, returns all information about each identity. Consider the following search query:

{
    "indices": [
        "identities"
    ],
    "query": {
        "query": "*"
    }
}

In my tenant this will return a response that is 2.25 MB. That alone will exceed the workflow limit.

To answer your question about limiting what is returned, you can do that by replacing your Get List of Identities action with an HTTP Request that uses the search API. Craft your query to narrow down the results to just the identities you need and add queryResultFilter to specify exactly which fields you need. For example:

{
    "indices": [
        "identities"
    ],
    "query": {
        "query": "*"
    },
    "queryResultFilter": {
        "includes": [
            "id",
            "name",
            "attributes.email"
        ]
    }
}

This query reduced my response size from 2.5 MB to 58 KB! This may resolve your loop issue.

[mosareini (Mohamad Sareini)]

@colin_mckibben Perfect. That is the exact solution I was looking for! Thank you Colin, but unfortunately this is a short lived W, as the loop is still giving me an error with that shortened results list. Here is the loop with the step output, any ideas on what I might be doing wrong?

[colin_mckibben]

“Unexpected end of JSON input” sometimes means that your http request is returning an error but the error message isn’t in JSON format. Double check the configuration of you http action in the loop. Try manually running them with the input you provided. If you can provide the workflow json, sanitized of any sensitive information, that would also help us troubleshoot.

[mosareini (Mohamad Sareini)]

@colin_mckibben I’ve run the logic inside the loop on it’s own and it works fine, it created the servicenow tickets and populated the data I wanted, but that was only for one identity per workflow execution and not a list of identities. I’ve attached the workflow json below, thanks again!

CopyofCreateSNOWDEVTicket20241001.json (3.5 KB)

[colin_mckibben]

I see that your “Get Identity Action” within the loop is referencing an HTTP request action outside of the loop. Steps within a loop cannot references steps outside of the loop. You can only reference the loop input or loop context.

For the Get Identity, you are specifying $.hTTPRequest1.body.id. You should specify $.loop.loopInput.id. However, I also think that the Get Identity step is redundant. You are already passing the identity information into your loop input. Just use that instead of calling Get Identity again.

[mosareini (Mohamad Sareini)]

@colin_mckibben I referenced that step outside the loop just cause I was testing to see if maybe passing a different step into the loop would help, and forgot to change it back before I sent the json file, my apologies. It was erroring out before I made that change. I also removed the Get Identity step, and the loop is still erroring. I can see that the loop input is correct, but for some reason the output is still giving me that end of JSON error. I downloaded the workflow execution details and noticed the only fail is here, I’m assuming this is the loop error that could not be parsed.

[mosareini (Mohamad Sareini)]

I’m not sure where the parameter looking for a secret is coming from, I am not using Oauth in the HTTP request inside of the loop.

[colin_mckibben]

Your workflow definition shows you have configured basic auth for your HTTP request action inside your loop. You may have better luck deleting the HTTP action inside your loop and creating it again. Sometimes secrets can be sticky and remain in the definition even after you switch back to no-auth.

[mosareini (Mohamad Sareini)]

That’s exactly what was happening. I noticed that parameter secrets/[secret] is how the http request outputs the password for basic auth. I just pasted the password into the HTTP request again, and it worked. Thanks again for all your help @colin_mckibben, I appreciate it.

[mosareini (Mohamad Sareini)]

@margocbain Thank you for this, I didn’t realize you could filter down the response using $.getListOfIdentities.identities[*].id . Colin ended up providing me with some json for the search api that filtered down the response body and I got the work flowin’ now lol. Thanks again!

[dominick-miller]

I am running into the same error message: “unexpected end of JSON input”.

For how much workflows are expected to be used, it is disheartening how difficult debugging inside them is. There is basically 0 visibility into Loops, for example, and this error message gives no indication on where to address the issue without searching the SailPoint forums.

[mosareini (Mohamad Sareini)]

Hey @dominick-miller , I absolutely agree. There is a good amount of bugs and or lack of features that lead to having to spend more time troubleshooting than I would like to. In terms of you not being able to see the errors, if you download the workflow execution details, you’ll get a csv with all steps in the workflow, and the result of each step. That’s how I’ve been looking into errors inside the loop. It will however, group the steps inside the loop into one line item, but you’ll be able to see the actual error being thrown instead of that “end of JSON” error. If you need more help locating the execution history let me know. Hope this helps!