When you change the groups for an user, either add or remove from them, for the effects to take effect that user must logoff and then login again.
So if you're adding a user to the administrators group, and the user is logged on, he must close session and login again.
Restart of the whole system is not required in any case.
