This is as designed. When creating the set from the selection points, it only contains those points. You can filter on those points selected.
I would still argue that edit of such sets should be allowed. Like this one could graphically select the bulk and then do the fine selection (adding or subtracting elements) through an edit. This way one could also select elements that happen to have empty coordinate values but useful plot information otherwise (see for example http://community.tableau.com/message/186394). Especially if one has many datapoints such an edit possibility would be important.