The All Users group exists in every site by default. Every user added to the server becomes a member of the All Users group automatically. You cannot delete this group, but you can set permissions for it. You cannot delete All users group in server. Please change the permissions of this group to none so that they will not effect the permissions.
Awesome thanks for that!