IdentityNow Workflows provides a loop operator that allows a workflow to process many items in a list of data in a quick and efficient manner. While the loop operator is a built-in component of Workflows, there is another way to achieve looping functionality using features that are currently supported in the product. In this blog, I will demonstrate how to implement recursion in Workflows to achieve looping behavior without the need for the loop operator.
What is recursion?
Recursion is a form of iteration that is achieved by creating a function that will call itself. To demonstrate this concept, let’s compute the factorial of a number using a loop.
num = 7
factorial = 1
for i in range(1, num + 1):
factorial = factorial*i
print("The factorial of",num,"is",factorial)
Now let’s compute the factorial of a number using recursion.
def factorial(x):
if x == 1:
return 1
else:
return (x * factorial(x-1))
num = 3
print("The factorial of", num, "is", factorial(num))
As you can see, recursion employs the use of a function that will call itself in order to perform the iterations necessary to calculate the result. In most cases, loops are easier to implement and understand, but recursion is an option in many programming languages that can make some problems easier to implement.
Recursive Workflows
Recursion in Workflows employs the use of the following three features to process a list of items.
A recursive workflow will continuously invoke itself until there are no more items left in the list. In this example, the workflow is started with a list of three numbers. It will process each number in the list in turn, and then invoke itself to process the next number. It will repeat this process until there are no more numbers in the list.
flowchart TD
Z[Invoke workflow with list of items] --> |"[1,2,3]"| A
A[External Trigger] --> B{If length of list > 1}
B -->|no| C[Perform actions on last item]
B -->|yes| D[Perform actions on first item]
D --> E[HTTP Request to invoke this workflow with first item removed]
E -->|"[2,3]"| A
Pros and Cons of Recursive Workflows
Recursive workflows have some advantages compared to the loop operator, but there are also drawbacks to using recursion. When possible, prefer to use the loop operator. However, recursion is a viable option if the pros outweigh the cons for your particular use case.
Pros
- There is an option to implement synchronous recursion, which allows you to execute a series of commands on each item in the input list in the order in which they are received. The loop operator is strictly asynchronous. Asynchronous is usually better, as it is much faster than synchronous, but in situations where you need synchronous execution, recursion allows for that.
- There is no strict input limit for recursive workflows. If you need to process more items than the loop operator can support, then recursion allows you to do this. However, it is a good idea to check the length of your input before invoking a recursive workflow to make sure you don’t accidentally iterate over a very large data set, which can have performance implications.
Cons
- Recursive workflows are harder to implement and understand, especially for less advanced users.
- Recursive workflows can only accept arrays as input, not objects. This means that they cannot work with a context like the loop operator can, which limits the use cases they can support. For example, if you need to revoke a list of entitlements for an identity, then you would normally pass the list of entitlements to the loop input and the identity ID to the loop context. However, since recursion can only accept an array as input, you can only pass it the list of entitlements and not the identity ID as well, which means this use case won’t work with recursion. Support for objects as an input type in recursion may be coming in the future.
Synchronous Recursion
Recursive workflows can be configured to run synchronously, which means the recursion will process each item in the list in the order in which they are received. For example, if the list [1,2,3] is provided to the recursive workflow, it will process 1 first, then 2, and finally 3.
Implementation
A fully implemented synchronous recursive workflow will look like this, at a minimum. This workflow may be larger depending on how many actions you need to perform on each item in the list. You can download the workflow script and follow along in your tenant. Continue reading to learn what each step in this workflow does, and how to configure it.
Synchronousrecursion20230818.json (2.6 KB)
External Trigger
Click on External Trigger and generate a new access token. Make sure to save the client ID, client secret, and client URL, as you will need these to configure the HTTP Request actions later.
Verify Data Type
The Verify Data Type step doesn’t require any further configuration. It is configured to check if the type of data coming from the External Trigger is an array or not. This check is necessary since Workflows will remove the array brackets from single item arrays. If the trigger input is an array, then it takes the right path, which will process the first item in the array and then pass the remaining items to the next invocation of the workflow. If it is not an array, then it knows it is at the last item in the list and it will take the left path to process that last item and finish the recursion.
Verify Data Type 1
If the External Trigger input is not an array, that means the recursion has arrived at the last item in the original list. We need to do one more check to see if the trigger input is null or not, which depends on the initial payload sent to the External Trigger.
If the initial trigger input is an array of one item (ex. [{"id":"123"}]), then the workflow will take the right path at Verify Data Type. It will perform the actions on the only item in the array, but it will invoke the workflow one more time with a null object, since $.trigger[1:] results in a null value for an array with only one item. Therefore, we need to check for a null value which indicates that there is no more input data left and the workflow should terminate. If there is a null value, then the workflow will take the right path of Verify Data Type 1, which will terminate the workflow.
However, if the initial trigger input is an array of more than one item, then the last item in the recursion will not be null. It will be a single object. Verify Data Type 1 will take the left path to process the last item of the recursion one last time, before terminating for good.
HTTP Request 3
If the External Trigger input is not an array and it is not null, that means the recursion has arrived at the last item in the original list. The only thing left to do is perform the series of actions necessary to process the item and then terminate the recursion for good. To keep this example simple, the only action we perform on the list item is an HTTP Request action to send the data to webhook.site (read more about webhook.site in workflows). In a real scenario, you would have one or more actions that process the item, or you might have a separate workflow that you can invoke via an External Trigger to better encapsulate the processing actions.
It’s important that both the left and right branch of the Verify Data Type step perform the same series of actions to process the data. There will be a lot of copy and paste. Just make sure that you use the correct JSONpath expression for actions in each branch based on whether the trigger input is an array or not. The JSONpath used in the HTTP Request 3 action is simply $.trigger since the input is not an array.
HTTP Request
If the External Trigger input is an array, that means there is more than one item left to process. The right branch of the Verify Data Type step is followed by a series of actions that will process the next item in the list. This could be creating a cert campaign, submitting an access request, to name a few examples. To keep this example simple, the action that will process the next item in the list is an HTTP Request that will send the data to webhook.site.
It’s important that both the left and right branch of the Verify Data Type step perform the same series of actions to process the data. The right branch will be an exact duplicate of the left branch, with the only difference being the JSONpath used to reference the input data. The JSONpath used in the HTTP Request action of the right branch is $.trigger[0] since the input is an array and we want to process the first item of the array.
HTTP Request 1
The last step in the right branch is to use an HTTP Request to recursively invoke this workflow with the remaining items in the array. Since the previous action(s) processed the first item in the trigger payload, we need to use the JSONpath $.trigger[1:] to send everything but the first element in a new invocation of this workflow.
The target of this HTTP Request is the External Trigger of this workflow, so you will need to configure the authentication and request URL to point to the trigger of this workflow.
Caveats
Synchronous recursion is slower than asynchronous recursion. The workflow must wait for all actions that process the current item to complete before invoking the next iteration. If the order in which items are processed is important, then synchronous recursion is the right choice. Otherwise, use asynchronous recursion for much faster execution times.
Asynchronous Recursion
Recursive workflows can be configured to run asynchronously, which means the order in which items are processed is not guaranteed. In many use cases, processing items in order is not necessary. For example, if you need to disable a list of identities the order in which they are disabled probably doesn’t matter. Asynchronous recursion is much faster than synchronous recursion since the workflow doesn’t have to wait for processing steps to complete before invoking the next iteration of the workflow. In most cases, you should prefer to use asynchronous recursion.
Implementation
A fully implemented asynchronous (async) recursive workflow will look like this, at a minimum. Notice the use of the loop operator in async recursion. Using the loop operator greatly speeds up the execution of the workflow. This workflow may be larger depending on how many actions you need to perform on each item in the list. You can download the workflow script and follow along in your tenant. Continue reading to learn what each step in this workflow does, and how to configure it.
AsynchronousRecursionLoopsDemo20230818.json (4.3 KB)
External Trigger
Click on External Trigger and generate a new access token. Make sure to save the client ID, client secret, and client URL, as you will need these to configure the HTTP Request actions later.
Loop
Since the loop operator processes items asynchronously, we use it in our async recursive workflow to process the input items asynchronously as well, greatly speeding up execution times. Since loops currently have a limit of 100 input items, the input to Loop will be the first 100 items of the External Trigger payload.
HTTP Request 3
Inside Loop is a single action; an HTTP Request to invoke webhook.site with the data in the current item, referenced with the JSONpath $.loop.loopInput. This is a simple demonstration, but in practice you will likely have more actions inside Loop to process each item accordingly. This could be disabling an identity, or starting a certification campaign.
Compare Numbers
The loop will process the first 100 items asynchronously. After the loop, we need to check how many items were in the External Trigger payload to determine how to proceed. Compare Numbers will check the length of the trigger input. If the length is greater than 100 (the limit of the loop step), then there may be more items to process, which will require a recursive call to this workflow with the remaining items in the payload. If the length is less than 100, then we know the loop was able to process the last of the items in the trigger payload, and we can terminate the workflow.
Compare Numbers 1
Since Workflows will remove the array brackets of single item arrays, we have to handle the edge case where the trigger payload contains 101 items. If we attempt to invoke this workflow with a JSONpath expression of $.trigger[100], then the array brackets will be stripped and the trigger payload will contain an object. This will cause an error, as the loop is expecting an array, not an object. Therefore, we need another compare numbers operator (Compare Numbers 1) to check if the trigger input is more than 101 items. If it is more than 101 items, then we can recursively invoke this workflow with the remaining items since the array will be preserved. However, if we hit the edge case of 101 items, then we just have to process that last item with the exact same steps performed in the loop and then terminate the workflow.
HTTP Request 1
If we have 101 items left in the trigger input, then we need to process that last item (item #101) with the exact same steps performed in the loop. In this demonstration, the only step that we performed in the loop was to send the item data to webhook.site. HTTP Request 1 will use the JSONpath $.trigger[100] to send item 101 to webhook.site.
JSONpath array indexing is 0 based.
$.trigger[0]points to the first item in the array, while$.trigger[100]will point to the 101st item in the array.
HTTP Request
If there are more than 101 items in the trigger input, then we need to recursively invoke this workflow, passing all of the items after the first 100, which were already processed in the loop. HTTP Request is configured to invoke the External Trigger of this workflow with the remaining input items.
The target of this HTTP Request is the External Trigger of this workflow, so you will need to configure the authentication and request URL to point to the trigger of this workflow.
Testing Recursive Workflows
Recursive workflows require the workflow to be enabled in order to call the External Trigger with the remaining input. Because of this fact, the “Test Workflow” feature is of minimal value. You can use “Test Workflow” to test a single run of the workflow, but since the workflow isn’t live it won’t be able to invoke itself again. Therefore, the only way to truly test this is to enable the workflow and invoke the External Trigger with small test payloads. You can view the execution results to see if it worked or not. You can use a tool like Postman to test recursive workflows. This will require you to make two API requests: one request to get an access token, and one request to trigger the workflow with test input.
Get An Access Token
Using the Client ID and Client Secret provided by the External Trigger of your workflow, make the following API call to get an access_token.
POST https://{your tenant}.api.identitynow.com/oauth/token?grant_type=client_credentials&client_id={your client id}&client_secret={your secret}
Copy the access_token for use in the next step.
Invoke the Workflow
The External Trigger will give you the Client URL needed to invoke the workflow. Copy it from the external trigger and use it in your next API call to send test data.
Then, make an API request to the Client URL with the data you want to test. In this demonstration, we will send a very basic array of objects that will be sent to webhook.site.
curl --location 'https://{tenant}.api.identitynow.com/beta/workflows/execute/external/{workflow_id}' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {your access_token}' \
--data '[
{
"key": "1"
},
{
"key": "2"
},
{
"key": "3"
}
]'
In this example, the workflow sends the data to webhook.site. After invoking the test payload, I see the following:
Using a Recursive Workflow
Recursive workflows can be used anywhere you would normally use a loop operator, as long as the loop doesn’t need a context defined. At this time, recursive workflows can only work with an array input since there is no way to pass additional information as is supported by the loop context variable.
The following workflow is a simple example of where a recursive workflow can be used. This workflow runs on a scheduled trigger that periodically queries the list identity certifications endpoint for any certifications still in the “Active” phase. Normally, a loop might be used here to check the created date of each certification to see if the certification has been active for more than a certain period of time, say 14 days. If a certification has been active for at least 14 days, then send a reminder email to the reviewer to complete the certification.
This workflow will fail if there are more than 100 certifications in the “Active” phase. However, if you use a recursive workflow to process the list of certifications instead of a loop, you will be able to process more than 100 certifications. To replace the loop operator in this workflow, you first must create a recursive workflow that performs the same actions that are performed inside the loop operator. Then, replace the loop operator in your main workflow with an HTTP request that invokes the recursive workflow with the list of certifications. Building the recursive workflow is an exercise left to the reader, but the main workflow should look like this. The first HTTP Request invokes the list identity certifications endpoint to get the list of active certifications, and the second HTTP Request invokes the recursive workflow with the list of certifications in the request body.
Debugging Recursive Workflows
Synchronous
Synchronous recursive workflows will create a new workflow execution for each iteration. As a result, you will see many executions that are close together in time. If any individual execution fails, you can click on that execution to see why it failed. For example, if I invoke the workflow with five items, you will see five executions very close together in time.
Asynchronous
Since asynchronous recursive workflows use the loop operator to speed up execution, the execution history will look different than for synchronous recursion. If you have an input loop of less than 100 items, then only one execution will be triggered, and the details for each item in the loop will be contained within that single execution. You will need to download that execution report to check each iteration of the loop for any failures, just like you would for a normal workflow that uses loops.
If you have an input of more than 100 items, then you will see one execution for each time the workflow was recursively invoked. For example, if you invoked the workflow with 230 items, then you will see three executions.
- The first execution report will contain the results of the loop for items 1-100
- The second execution report will contain the results of the loop for items 101-200
- The third execution report will contain the results of the loop for items 201-230
Stopping infinite loops
If you don’t properly handle the termination conditions of a recursive workflow or correctly pass the remaining list items to the next iteration of the recursion, you may end up creating an infinite loop, which means the recursion will never end and just keep calling itself. If you encounter a scenario like this, the correct course of action is to disable the recursive workflow and wait for all running executions to finish. Once there are no pending or running executions left, you can then fix the workflow so that it does not exhibit infinite looping anymore and then enable the workflow again.
If you manage to get it working, would you mind creating a colab and/or blog post sharing your implementation?