It seems that multiple clients have issues where the VA doesn’t have an up to date flatcar installation (old version of kernel/docker etc). You can check the version using less /etc/os-release.
If you are not using at least on of these version, you may want to contact SailPoint Support to see why the update of the base operating system doesn’t work.
Another issue that you may be affected by (but which may not really matter to you), is that log rotation isn’t working.
Check which files you have on the VA under /home/sailpoint/log: are they huge? Should they have been rotated by now? Contact support as rotation may not be enabled…
Thanks Andrei.
Do you know if Sailpoint VA(flatcar) releases and updates follow the Official Flatcar release-schedule?
And is there somewhere SailPoint announces these updates?
Thanks for mentioning this. A Client just brought up today that their Logs are not rotating and saving the backup as expected based on KB0011188.
I have advised them to contact Support, but is there anything they can check in the short term? Since the backups are not being saved, they are unable to get accurate logs for debugging.
Hi @mkjn1, I don’t think they follow the official Flatcar releases automatically. I don’t know when they decide to update to a newer version, and if they skip some versions.
I believe:
they should make this process more transparent, as we are talking about security updates
they should have an automatic check of the version installed and notify you (email/ISC interface), if the version is stuck to an older version
Just for reference, deleting the machine-id and triggering another update managed to update from a Flatcar version from beginning of 2023 to a version from January 2024.
They may try to enable the logrotate.timer. It was not possible to execute this command on the VA, but after an update I was able to execute it. I hope/guess they rolled the change to everyone.
Thanks, I’ll have to look. The file is getting rotated, but the previous one is not saved anywhere, making it difficult to debug issues near when it rolls over. Support said that the KB Article I referenced above was actually incorrect (and has since been updated) in stating the logs will be backed up locally.
So what I am looking for is a way to get the current CCG.log file, and at least the one previous to it so we can better review issues. I am not sure if updating the log4j2 file referenced in the logging document also allows for updates to the log configuration, or only the loggers.
This means the file is overwritten, not that it gets rotated.
The rotation happens outside of Log4j/CCG Java code, there is a systemd task triggered by the logrotate.timer.
@adamian I am seeing the LogRotate in my Test Environment VA, but when I look at the sailpoint configuration in “/etc/logrotate.d/sailpoint”, they have it set to “rotate 0” and the prerotate task is set to truncate the log file. We are unable to edit this file also as it is set to read-only.
The Flatcar base image was finally updated yesterday from the January version to something more recent (the version from April), so it is now only one version behind (the May 22th version is missing).
Hopefully the issues are fixed now and new Flatcar versions will be promptly used.
Latest update from SailPoint regarding updates of the VAs:
I hope you are doing well. So further as of now, we don’t have any actual details, as per the DevOps team, the logs provided don’t show any as such as well, but could be estimated a kind of delay in the updates could be some possible delay over the VA or something from the flatcar side, which is not providing any ultimate root cause, unfortunately for this scenario, However, I hope this would have resolved your concern by updating.
@questjj , @hranjan3, sorry to hear that SailPoint didn’t address this issue yet, although it was reported for months and their documentation says something to the contrary.
Try to call this command: sudo systemctl enable logrotate.timer
It should say: Created symlink /etc/systemd/system/timers.target.wants/logrotate.timer → /usr/lib/systemd/system/logrotate.timer.
We didn’t have permission to execute the command at first, but hopefully this is allowed for all VAs now.
Hi @adamian Thanks for your response. I tried the given command but its not working with my access level.
systemctl enable logrotate.timer
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-unit-files ====
Authentication is required to manage system service or unit files.
Authenticating as: root
Password:
Failed to enable unit: Connection timed out
I think its trying to authenticate using root account only.
yes right it worked now. I can see message “Created symlink /etc/systemd/system/timers.target.wants/logrotate.timer → /usr/lib/systemd/system/logrotate.timer.”
I have few queries - will log rotation start from now onwards, how it will calculate rotation time like day wise or hour wise. Do we have any configuration where we can specify how we want log rotation to happen.
If for some reason a VA won’t update, here is the series of commands that we got from SailPoint to force the updates.
1. Login to your VA and run the following:
o cat /etc/os-release
§ This will give you the base OS version; keep a note of it so you can compare later; it'll look like "VERSION=3602.2.2"; if you see 3815.2.5 or higher, there's no need to update that VA. You'll find the current version here: Releases | Flatcar Container Linux
o ssh -V
§ This gives the SSH version. You'll find the current version here: Releases | Flatcar Container Linux
o wget -O ~sailpoint/stunt_22.sh https://raw.githubusercontent.com/sailpoint-oss/colab-stunt-script/main/stunt.sh; bash ~sailpoint/stunt_22.sh -u
2. That process will download the stunt script, run it, and ask if you want to reset the machine-id. Press "Y" and hit enter/return
3. The update script takes a few minutes, during which time your VA will appear to be doing nothing; please give it at least five minutes before canceling the job or resetting the VA (you shouldn't need to do either)
4. Once complete, hopefully you'll see a colorful message (red and yellow) that says you need to reboot. Do so with this command: sudo reboot
5. Wait approximately 30 seconds and log back into the VA
6. Run: cat /etc/os-release
