Are you using 32 bit or 64 bit Windows? If 32 bit then using the /3G boot.ini switch will let Tableau use a little more memory.
The biggest memory usage comes from trying to display a lot of data at once, for example a very large crosstab type display with millions of rows. Connecting to a large data set whether an extract or live connection should not use much memory.
There can be a lot of reasons for running out of memory. I suggest contacting Support and sending them your logs and workbook with extracts if possible. They will be able to help.
What is this boot.ini switch?
On a Windows 32 bit O/S you can include an entry which I think is actually /3GB (as opposed to /3G). That tells Windows to allocate 3 GB out of the 4 GB of virtual address space to applications and only 1 GB to the operating system. By default the address space is split 2 GB for system and 2 GB for user space.
That allows Tableau to use more memory for its internal data - which translates directly into being able to handle bigger result sets and display more marks before running out of memory.
Note that this is the split of virtual address space. In practice, I have found that this increases the maximum size of the actual Tableau application by a factor of about two (I measured it at 1.2 GB increasing to just over 2 GB when I used it).
By way of comparison, running Tableau (which is a 32 bit application) on a 64 bit windows platform increases the maximum size of the Tableau executable to close to 3.5 GB - so that is the way to get the most available memory until such time as Tableau has a 64 bit version available.
Note that using the 3GB switch can have undesirable consequences. Only having 1 GB of memory available for the operating system may break other things. In my case when I first used it the result was that the display driver wouldn't load - which meant my laptop fell back to VGA graphics (640 x 480). So I could now display over 1 million marks - but at unusable screen resolution.
But don't despair - there is another switch (which I've forgotten but will dig out later) which lets you give a little bit of the 3 GB back to the operating system. That was just enough to let the display driver load for me - and it then worked very well. I vaguely recall that there was some other minor side-effect, but can't remember what at the moment.
I dug out the parameters I used when I was doing this:
The value of 2900 worked for me - but you might need to experiment.
This microsoft article explains all about it: