You have the option to stay on 10.4. It will be maintained for a bit. I believe Tableau maintains the previous 3 or 4 major versions. Eventually your customers will want all the new features in 10.5.X then you'll be forced to upgrade.
When you go to upgrade (10.4 to 10.5) you will upgrade Server and Desktop so there's no real problem. Hyper is better than TDE, in 10.4 TDE is the only option and in 10.5 Hyper is the only option. The release notes are written like that so people realise that they can't downgrade.
My advice would be - sit tight, upgrade when you are ready, but the same with every upgrade you will have to upgrade Desktop & Server versions at the same time.
Maybe I'm missing something, but I don't think you have to upgrade your 10.4 files. We are testing this on a 10.5 server right now and we haven't found anything wrong with them.
In the event you wish to upgrade your viz to 10.5, then of course it will update itself but there isn't any impetus to change if you don't want to.