The groups themselves should not be an issue. Aggregation will query which groups User has and bring its member list across, instead of bring the group and all its membership across in one go.
FYI: I would some caution on how to build the IT Roles. IT Roles are a great abstraction to bundling up multiple access into one container. If you are building them with a 1:1 entitlement to IT role ratio then I would advise not to do this, and just leave the entitlement as-is…