We have a distributed server environment and so far performance has been great. The distributed environment is 2 x 4 core virtual wintel servers with 16 GB of memory on each. We are actually looking to double capacity and are now looking to see if it's best to go with a virtual distributed environment or a single physical server with the # of cores and memory being equal among the two configurations.
The only downside I've seen is that the distributed server requires that the non-primary server(s) are workers, meaning you often have to upgrade the worker separately when moving to a new version. That's about the only downside though.