I apologize in advance for the long question:
We currently have an extract in our Tableau Server environment that has about 60 million records, with the size of the extract being about 20GB. There are absolutely no issues with the performance.
This extract contains a great bit of financial details of various products. There is a need for categorization of these products for various analyses, which we implemented via pre-created filters("Extended Annuity Products" where type =Annuity and some other condition). So, if they drag "Extended Annuity Product" into the filter area, the data is limited to "Extended Annuity Product". This table is HUGE and we have about 50 types of pre-created filters for users. So, if they don't use this filter, they will get information about all products. This is essentially supporting a lot of analyses in the company.
Our company recently purchased another product that harmonizes creation,maintenance and use of these various product categorizations. There are millions of products identified by a unique key along with other metadata. So if someone goes to this new system and creates ("Extended Annuity Products" where type =Annuity and some other condition), the output of this will be sent to our system at the lowest level - so it will contain two columns - one with the name of the category and one with the product id of the product. There could be hundreds of thousands of records under each categorization
This will mean we can't use filters anymore for product categorization and have to use the output from this new system which contains the categorizations. We can take our table and join on productid that will give us the Product categorizations we want, but will significantly blow up the size of the extract and potentially effect the performance.
I am not a big expert of Tableau, but can experts provide some insights into how this can be potentially handled? We will be working with some consultants to help us with this, but I wanted to see if we even have any options other than creating a huge table and extract.