One way to do it: we need to check for the top 5 in a different way for each state. Here we use a calculation looking at index()<6 to hide any items lower on the list. This is set to ignore sub-category and look across the state:
Hang on, my sort is wrong - we need to sort using a set. Back soon with proper answer.
I have found http://kb.tableausoftware.com/articles/knowledgebase/finding-top-n-within-category to be extremely helpful in doing exactly what you are looking for. You might have to play with it a bit to get it perfect, but follow that guide and you should be able to get the top 5 items per department.
1 of 1 people found this helpful
Jeff's method has the advantage of explicitly setting the sort whereas mine relies on you doing it from the pill (e.g. you can get bottom 5 by just changing the sort order on mine)
Edit: looking at that method, you could explicitly set it. They actually don't, relying on sorting via the pill menu.
Edit 2: Ok, so this KB was probably created before the introduction of index(). However, it does get right what I didn't - that is the proper nested sort.