When you use a data extract, we only load columns into memory that are necessary to answer the question being asked, so the answer to your question is "no". Having folks connect to an extract directly over the network is generally a bad idea, however. You want the file local so we can get in and out of it as quickly possible and so that Tableau isn't at the mercy of your network.
If your database gives you very poor response time, then Tableau will also give you poor performance UNLESS you create an extract off of the view in SQL Server. There's no way for Tableau to "guess" at what the resultset that is coming back from SQL Server will be, so we need to wait for the query to return results to us.
Hope this helps.