Can you share a workbook? Hard to tell from this description.
Actually I figured it out. I had another field on Detail and this caused it. I had it there originally because it is used in my filter action but I didn't want it on columns or rows. Once I removed this behaved, however, I couldn't find another good place for that same field I want to use in my filter action. Can't hide it on columns or rows, nor use as an attribute.