I created a custom color palette for use in my workbook (10.5.5) and it works perfectly fine...
until I publish to Tableau server (also 10.5) when it switches back to the default colors. I am using published data sources if that makes any difference but I still have no clue why my palette is switching.
The colours are taken from the published data source as that is how you're workbook is connecting to the data.
Have you published the data source to server, and have your workbook connect to the published data source?