For anyone who comes across this with the same problem.
I tested trying to setup the group schema & group aggregation with a group membership attribute, but couldn’t get it working.
In the end, I had to create a custom rule to pull all the groups and members, store the entire collection in the transient values attribute on the connector and then compare each user aggregated to the members list of each group stored in that transient values attribute.
Info on the transient values attribute here: Cutom object to store data for reference - #4 by naveensingiri419