we are using server 10.1 and are (so far) controlling all permissions on the server side at the project level. As far as I Can tell summary data permission controls the crosstab and the summary data. you can disable full data download from the project permissions page. From what I have seen so for in my workbooks, the summary data and the cross tab contain the same data, just is a slightly different format so just disabling the full data download should work for what you need.
On another note, if you publish data sources (specifically extracts) to tableau server, any user that has data source "Connector" privileges can connect to the data source with tableau desktop, and thus see all of the data. I have read mixed information on which permissions need set in order for users to utilize views. I currently have most users set with no Data source privileges, for the most part they can still access views, there are a couple that are giving data source permission errors, other views using the same data source have no issues. I currently have a case open with support on the issue. I only mention this here because it's something I didn't think to check until I read it in a forum, and it sounds like something you will want to avoid as well.
Hope this helps