The ways you can filter in Tableau are via:
- extract filters (applied when an extract is created/updated)
- data source filters (applied to every query, no direct end-user control via web interface except via a parameter)
- filters (fields on the Filters Shelf)
- quick filters (fields on the Filters Shelf that display selection widgets)
- parameters (which are tied to fields used as filters or data source filters)
- Filter Actions
- URL parameters
If your custom logic can be expressed as a Tableau calculated field or an R script then you can embed that inside Tableau and use the existing filters or quick filters. Otherwise you'd likely need to use the JS API.
Last I knew, the REST API is more about administering Tableau (publishing workbooks, setting up permissions, etc.) and not so much about interaction with a viz.
thank you for the expressive answer, that's what I call a informative answer!
All I have learned about the JS API is I can use filters only as a, what you call, Tableau calculated field. Am I missing something?
To complete Jonathan's answer, you can also use R programming language to define calculated fields (=custom logic that you have written) and use it as a filter.
If you can give some details on your use case then we might be able to give you less generic answers