This may not address all your questions but at least to start the process:
(1) Not sure there's necessarily a "rule of thumb" but definitely not a bad idea to have at least a "pool" of Tableau Server Administrators available. For example if folks are out sick or on vacation then you'd still have resources available. The same could be said if a Tableau Server Administrator decides to leave an organization this can be painful if they were the only one.
(2 & 3) Tableau Server does most of its operations with interacting with the customized Postgres database it comes with. That said, it does need some periodic interaction to keep things running well. You may want to have a look at the following forum link to identify "stale data"
One item folks may not think about but if you allow workbooks to collect on Tableau Server, if they involve extracts those extracts become part of the nightly backup file (*.tsbak). Extracts are compressed before they become part of the backup file but there is a "temp location" they all get copied over to before that happens and we've seen were some customers run out of drive space on their primary because it can't copy all the extracts over to be able to create the backup file.
If dashboards are really poorly designed this will tend to show up in how long it takes to load the dashboard. Those that consistently show a high render time would be good candidates to revisit to see if any optimization can be done and if best practices were followed. We have a whitepaper on this very topic: Best Practices For Designing Efficient Tableau Workbooks
(4) As you mention performance can vary depending on the number of users that are hitting the system at the same time. If your Tableau cluster doesn't have a dedicated backgrounder some things that can be done would be to schedule long running extracts "off hours". Also, it would be better to have the bulk of Subscriptions run prior to the beginning of the workday if the data is available.
My comments below.
Kirstin Lyon wrote:
1. what is best practice for number of server admins on a server? I would have thought around 3-4 - but looking for opinions.
Two Tableau Server admins is the minimum for the reasons Mark gave. I have three (including myself). The third one doesn't really have the skills but is the backup of the backup so in a bad-case scenario they can get a qualified admin added.
2. I'm assuming that I should have a policy in place that makes sure that we don't just archive content, but old content is removed regularly, Does this actually help with performance? or is it a way to help people find what they want?
Most definitely remove unused content! Our policy is to delete content that goes unviewed for 4 months. Normally I'm the one who actually deletes the content so I download the workbook and save it to a network drive as a "just in case" measure first and then actually delete it from the server. That network drive content then gets deleted after 12 months.
You want to remove unused content because it consumes server resources. If a workbook that is on an extract schedule and the workbook is no longer being used then CPU and memory resources are being used and thus taking away performance for other processes that could use those resources. Not to mention that you would eventually run out of storage space.
3. What is the cost of having too much content on a server (if any), and what happens if 90% of the content is poorly designed. Does that affect the content that is well designed?
Poorly designed content can consume more resources (CPU time, memory & hard drive space) and thus negatively affect ALL server processes, including the well-designed content. This becomes noteworthy during extract refreshes and view rendering times.
4. I'm also assuming there is a cost for if there is a time of day where a lot of people visit the server. Is that right?
Correct. Everything performed on the server consumes resources which in turn can affect performance (extract times, rendering times, subscriptions emailed on time, etc.).