Has anyone attempted the idea of a staggered wait on http requests in a loop. I’m trying to create ITSM tickets and the error I get is that if ISC send the request at the same time it will fail, the ITSM attempts to give each ticket the same ticket number on the backend and fails.
I have tried a simple wait (1 minute) on failure but if there are many failures then there is a chance it will fail again.
The error I get is as follows:
“workflowErrorMessage”: “Create ITSM ticket Err: task failed: activity error (type: sp:external:http:v2, scheduledEventID: 58, startedEventID: 59, identity: 1@sp-workflow-worker-prd-us-east-1-64c58f4bf6-wqmmv@sp-workflow-engine): request failed (type: HTTP Response Returned a Client Error, retryable: false): request failed: 400 - 400 Bad Request - {“ClassName”:“System.Exception”,“Message”:“A problem occurred when updating the record. Error Message (I): Violation of PRIMARY KEY constraint ‘aaaaaFaults_PK1’. Cannot insert duplicate key in object ‘dbo.FAULTS’. The duplicate key value is (75161).\r\nThe statement has been terminated.”,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:null,“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:null,“HResult”:-2146233088,“Source”:null,“WatsonBuckets”:null}”,
Checking with the admin of the ITSM they validated that this is a known issue by the vendor when simultaneous API calls come in. To add more context this workflow has been operating for months with minimal issues, but the workflow is expected to run much larger loops in the fall. I fear we will see these simultaneous ticket creation errors more often.
I am not sure if there is a current built in function that will support what you are looking for. I would recommend checking out this post:
If I was attempting to solve this one, I would use the API referenced in this post (or a similar one I can put in our cloud), and setup a random wait time.
HTTP Request - Retrieve random Number
Define Variable - Add the random number to the current time
Wait - Use your variable to define when you should wait until.
It still has a risk of having the ticket creation at the same time if the numbers are too close.
How many loop executions do you anticipate having at the same time?
That is a good reference! I’m not sure why I didn’t spot that one in my searches.
Max number at this time is in the mid hundreds, the only way an error occurs is when the requests are incredibly close together, we have not encountered this issue outside of SailPoint workflows. Most other loops we use are not simultaneously processed.
It’s important to note that the loops in a workflow process asynchronously and could be direct causation for the failures you are seeing. As opposed to using a loop and random wait times, have you considered implementing a recursive workflow?
3, 6
He who does not trust enough, Will not be trusted. — Lao Tzu
You may rely on it