In our Active Directory connector, we have defined after-create and after-modify rules to handle some business logic that ISC cannot. We used the example provided by Sailpoint as a starting point, and for the most part, this works well. The issue comes when we have a large number of people starting on the same day. If we have more then 20 or so people being provisioned, then my IQService server gets overwhelmed. In the after-create script we talk to AD, Exchange and SQL, and when so many instances of the script are spawned, not only does the IQService server slow down (understandable), but the Exchange servers start throwing errors about using too many connections.
I was thinking of changing the script so that instead of each instance of the script attempting to do the work, the script would instead connect to a SQL server and put the JSON into a table, then shut down. Every few minutes a scheduled job would start up, look for new records, process them and clear them from the table. This way one job opens one connection to AD, Exchange and SQL and processes multiple tasks. It seems more efficient and since the native scripts will complete and release resources quicker, the IQService server should run more smoothly.
I am wondering what the downside, if any, would be to basically queuing up the events and processing them in batches rather than doing them as they are generated.
Thoughts? Opinions? Dinner Ideas? Vacation Recomendations?
So, while the solution you described could certainly be viable you should be able to handle more than 20 people onboarding in one time with the existing infrastructure. I’ve seen a much larger scale than that - though they certainly needed some performance tuning.
What kind of specs are on your IQService machine? Can you describe a bit more of the actions that are required? And is all the logic in your After Script - or is it kicking off additional scripts?
It might be 20 people, or it might be 30, but it is somewhere around there. It does not happen every Monday, but once or twice a month, I get a handful of people that have an AD account, but that AD account is not configured correctly and the email to the manager does not go out.
The IQService box was provisioned based on Sailpoints recommendations. I believe it has 4 CPUs and 16 GB RAM.
The script does not call other scripts. It takes the information passed and checks to see if that person is already in AD based on the employee ID. If so, it logs the collision and stops. If there was no collision, it checks to see if they have worked for the company before (SQL query), and if so, re-uses the previous ID and email address. If not, it generates a user ID (using some unnecessarily complex logic, but it is legacy, so it had to be reproduced) and then checks to see if that user ID has ever been used (SQL queries) and will try again until it hits a unique ID. This usually does not take more than a second or two. Renames the AD account (IDN creates it using the employee ID) and sets the password to a default value based on the attributes of the user. Then it looks at other attributes to determine the domain name to use on the email address, provisions that mailbox in Exchange, sets the UPN to match, composes an email to the manager to let them know the user ID and email address along with instructions on how to calculate the default password. I sampled the logs that the script generates and it looks like it takes less than 15 seconds per user to complete.
I agree that the number of users you are seeing shouldn’t have an impact. If number of concurrent connections are an issue, you could go the route of an SQL database, but a more minimal, less infrastructure solution could be to push the users to a CSV and then use a scheduled script to process those users with only a single connection.
Just an FYI, in my experience with writing to a log in the “after create” script is that the IQService is launching these instances in parallel so there are issues with instances opening the file for writing and blocking other instances. The solution is error handling, waiting a random time and retrying, or using different filenames. Having all instances writting to a single CSV would be a problem. Would be nice to have the option of having IQService launching one instance and passing all of the users as input instead of spawning one instance per person.