No, not really. Whichever data source you "use first" in a view (by dragging) becomes Primary. There's no way to "swap" afterwards except start again and drag a field from the OTHER data source first.
Agree with Sai.
When you drag any field to your view it from your data source it becomes your Primary data.