Firstly, I hope this is not a self-signed cert as this would not be recommended for SSL for production systems.
However you have not reached that stage yet. For some reason Tableau was unable to read your key or cert file into it's key store. The usual reason for that is that the key file is still password protected. There are log files that can give you more information (usually vizportal-<n>.log) but please confirm first that you have removed any passwords from the key file and that you are meeting the requirements described in the help docs here: http://onlinehelp.tableau.com/current/server/en-us/help.htm#ssl_config.htm