I got it worked out. The hyphen was one issue, the other was I didn't properly quote my path to the registry file. I also modified the script to switch to the proper folder prior to running the initial command. My working script is as follows:
cd "Tableau Server\10.3\bin"
start/wait tabinstallck.exe -activate *product key*
if %errorlevel% EQU 536871018 (ECHO TABLEAU_ERROR_LICENSING_ACTIVATION_INVALID_KEY)
if %errorlevel% EQU 536871012 (ECHO TABLEAU_ERROR_LICENCING_GENERAL)
if %errorlevel% EQU 536871026 (TABLEAU_ERROR_LICENSING_UNLICENSED_EXPIRED_SUBSCRIPTION)
if %errorlevel% EQU 0 (ECHO SUCCESS)
tabadmin register --file "D:\*path to file*\*filename*.json"
Thank you for your help.