Attached is an example using the RANK() function. It's similar to INDEX() in that both are table calculations and so you can specify the scope in which they run. I suspect on your workbook the scope is what's causing the undesired behavior.
The calculation you can look at is called [Rank - SubCategory] and is defined as ([Top N:] is a parameter):
RANK_UNIQUE(SUM([Sales]), 'asc') <= [Top N:]
You can drag the calculation to filter and specify the scope here:
Once there, you can set it to run within each category like this:
That should do it!
Rank.twbx 435.0 KB
Thank you for your reply and suggestion, I think this should do the trick. However, I am running Tableau 9.3 and it will not allow me to open your provided example. If it is not too much trouble could you repost with a version for 9.3.