This would happen on your assignment criteria. Roles are generally auto assigned based on HR data. In order for this to be removed the user would have to not meet the criteria. Even if you remove as long as they qualify for access IdN will re add them to the role.
If you have use case where users should not get birthright access then you will need to figure out how to identify these users and add criteria to exclude them.
In that case I would review the workflow trigger and see what is triggering the add event. It sounds like you have a endless loop and need to figure out why the add is being triggered.
If you want us to take a look at your trigger please take screen prints of your triggers and post here please be sure to remove any sensitive data before posting.