One idea is below.
And change the filter to context.
But anyways, Reference lines format is not good, so I recommend that you use dual axis with gantt instead.
Remove reference line.
when you choose 4.
when choose 3.
What if you use a calculation like this one and drop it on label instead of SUM([Sales])?
You could use something similar for the reference line, although if you select more than 5 records the line would disappear. To avoid that you could use a dual axis.