This kind of issue can happen when a database driver reports that a particular field (even a simple measure) takes up a large number of bytes. When multiplied by the total number of records, this can add up to an out of memory problem which Tableau can anticipate in advance. The problem here may be that the driver's report for the size of the field may be incorrect, and issues like that are difficult to diagnose. I recommend you contact Tableau Support to begin investigating this issue.
In the meantime there may be some things which you can do to work around this problem or gather more information about the nature of the problem.
- Does the Custom SQL statement produce any fields which are binary in nature, such as BLOB or VARBINARY? How about other potentially large fields, such as CLOB, TEXT or a very large VARCHAR field?
- Can you reproduce this problem if you connect to Netezza via generic ODBC (i.e. the "Other Databases (ODBC)" connection option)?
- Have you verified that you have the correct Netezza driver? Tableau requires an older version of their driver due to incompatibilities introduced in their newer drivers. The Tableau Support team can help detect if you have the wrong driver.
I hope this helps,