The only relevant information I could find related to that specific error was for installations using SAML and 10.0 Tableau Server. Does that apply to you? If so, then this known issue on the release notes page describes the open defect our engineering team is currently working on:
After upgrading to Tableau Server 10.0, if you tried to configure Tableau Server to use SAML and specified the HTTPS protocol in the return URL, without configuring Tableau Server for SSL (i.e. SSL is being offloaded), an error would occur and configuration could not proceed: "invalid protocol specified for SAML return URL. Use https if SSL is enabled or http otherwise"
Were any other changes introduced recently to the Tableau Server, other than the config changes you listed?
If the above doesn't apply to you, I'd recommend rebooting the Tableau Server machine to see if that allows the processes to start running again. If it still won't start, I'd suggest backing out the changes by restoring a recent backup (.tsbak) with your old configuration settings. If none of that helps, file a support case to get more eyes on the problem.
Please provide more information. did you check the tabadmin.log file. Is there any Antivirus is up and running.
Thanks & Regards,