Try to ignore the Domain Users group from “Remove all access“ by using How to Filter Out Domain Users Entitlements on Remove All Access - Identity Security Cloud (ISC) / ISC Discussion and Questions - SailPoint Developer Community and see whether it works for you.