Edit the domain by using the
tabcmd editdomaincommand which you have mentioned
will work if the user name with which you are currently signed in exists in both the current domain and the new domain.
If the user does not exist in both domains, follow the below link.
Follow the steps in Modifying Tableau Server Authentication Method.
From how I'm understanding it, the function is for renaming to a new, unused name. Since "ab" already exists the new nickname isn't new. However, the error you get is very unfriendly so I would recommend submitting a Tableau Support ticket.