We are looking to automate the deployment and removal of virtual appliances within our AWS Virtual Appliance cluster. Our current VAs are years old and regularly have issues where support ask us to replace them, but we currently don’t have any mechanism for doing so without days of laborious effort. We are looking to completely automate this process so that it can be done quickly and even automatically where required.
I have a couple of questions relating to the virtual appliance:
The AMI we have is many years old. My understanding is that we need to request the latest AMI each time a new AMI is released. Is there a way we can always have the latest AMI available to us automatically?
Is there a way to “safely” shut down a Virtual Appliance? What I want to do is have the VA complete any ongoing provisioning/aggregation tasks, but not take on any new tasks, and then have some way of validating when all those tasks are completed before actually shutting down the VA. Is this possible?
I asked around internally and no one has attempted to automate VA deployment. You and I will have to be trailblazers on this one.
For point 1, you could download the latest virtual appliance package, which is listed in the “New Virtual Appliance” page. and then write a script to automate the deployment of this VM image to an EC2 instance along with the setup steps and configuration file. I don’t know how often the URL of the virtual appliance image is updated, so you will want to have some logic in there to alert you when the URL no longer works so you can update it.
Appreciate the response, I thought that might be the case.
I’m happy to be a trailblazer in this instance, also happy to share any lessons learned here.
Is the SailPoint provided AMI literally just this VMWare image converted into an AMI? I would expect that there would be some other changes to the image and would rather use the SailPoint supported AMI for deployment in AWS.
However if you can confirm the image is identical then this could be a reasonable approach.
We don’t do any manual migration. The VAs we are running now are the same EC2 instances as we initially deployed. The primary purpose of implementing this automation is to give us a mechanism to be able to replace these VAs in a low-risk manner.
SailPoint does regular ccg upgrades on our VA cluster. What mechanism do you use to ensure this is done safely without impacting our provisioning activity?
I’m asking our devops team about the AMI. What I know so far is that devops shares the latest AMI with customers, so there is no repository for you to download the latest AMI. They did say that once an AMI is deployed, there shouldn’t be a need to upgrade to a newer AMI. I’m meeting with devops next week to understand more about that. Can you share any details about why support needed you to upgrade your AMI?
Also, I did some reading about our VAs and how updates are deployed, which you can read about here. It looks like you have to have at least two VAs running in your cluster, both in a good state. The CCG upgrade requires a reboot of the VA, so it will only reboot one VA at a time to avoid any downtime in provisioning. I’ll ask devops more about this process and how this upgrade process ensures that no queued provisioning activities are lost during the upgrade.
I believe the AMI will automatically update to the latest version, but that kind of defeats the purpose if we’re always using outdated AMIs. It would also hurt the performance, since the VA would be up quicker if it was the latest version.
Honestly while it would be ideal to always have the latest AMI, I’d be more than happy to just be notified whenever there’s a new image, or altneratively just have a way we can update it every couple of months.
Yes we will always ensure there is one VA up and running at any one time, but it’s quite possible that if we have three VAs up all three might be doing something. It would be great if I could prevent one of those VAs from picking up new tasks while completing their current tasks.
I spoke with a devops engineer about the AMI. The AMI has a lot of configuration baked in to make it easy to deploy to AWS. There’s no documentation on how to use the vmware image to build your own, so you’ll have to get the latest from support. There currently isn’t a mechanism for automatically making the AMI available. You’ll have to contact support for the latest.
As for how to deploy a new AMI, you should perform the following steps:
Obtain the new AMI and deploy at least two EC2 instances with the new AMI.
Configure the new instance and make sure they successfully connect to IDN.
Shutdown each of the old instances one at a time. You may receive a failed aggregation alert, but this shouldn’t be a cause for concern. The new instances will pick up the load during the next aggregation.
Thanks Colin, I guess we need to just regularly ask support every couple of months for the latest AMI then? That’s not the best outcome… Any chance of some process improvement here?
Regarding your process, I’m a bit concerned about “step 3”
Shutdown each of the old instances one at a time. You may receive a failed aggregation alert, but this shouldn’t be a cause for concern. The new instances will pick up the load during the next aggregation.
So reading between the lines here, you do not have a mechanism for safely shutting down a VA, instead you just shut it down unsafely and let the job fail? Given SailPoint are in charge of when our VAs update, I’m a bit concerned with this response. How many failed jobs do we have as a result of SailPoint shutting down a VA in the middle of a provisioning task? What happens if a provisioning task is partially completed and the VA is unceremoniously stopped? Is there really no mechanism for ensuring jobs are finished before a VA is taken offline?
Aggregation jobs are not a particularly big concern, although some jobs take 4 hours or so and that failing partway through can cause trouble, but if a provisioning task is interrupted who knows what the result could be.
To clarify on your concern about the shutting down of VAs, the regular VA update initiated by SailPoint does not require a restart. The only time a shutdown is necessary is if you are decommissioning a VA, as is the case when deploying a new AMI. This decommissioning could be done over a weekend to reduce any impact caused by a failed job.
I believe the ccg service is restarted which amounts to the same thing, does it not? Unless you tell me you manage to upgrade the service without restarting it in which case I would love to know how it is done
