Currently the ability to add and remove users to Tableau Server requires a site or server role that will also grant access to view the workbooks on the Tableau Server. The users would have to at least be a site administrator which would give them access to that site.
Having additional permission models would be a great Idea to share in the Tableau Community Ideas space. These are reviewed by the Tableau Product Mangement team for upcoming releases.
Hope this helps!