I ran into the same issue and figured out a quick fix: use tabcmd to try to import the groups.
tabcmd syncgroup "domain\group"
Hope this helps.
To work around this problem, we used:
tabcmd synchgroup "domain\group name" -- site "site name"
Not sure why it fails using regular group web page on Tableau Server, but works using tabcmd. Still interested to know what log file can help us in future with these type of errors.