A silly question, I've written a web data connector to fetch data from a Project Haystack-based server, and so far, so good, as a prototype it works. However, as it has no concept of what data is already present in the Tableau server, it endlessly re-fetches the same data. We're after something of a data pump that continually fetches new data from the source server and pumps it in to Tableau.
Looking at the web data connector API, I see no function that lets me query what timestamp the last record is for a given column name. I see mention of a string: lastRecordToken, however it is not mentioned whether the value is persisted between executions on the Tableau server.
The other option is to use the Python SDK with pyhaystack to query for the data. No problems there, except once again, I don't see a function to call in order to find where the data is at. So I still have no idea where to start reading from.
Is there a way, given a data source and column name, to query what the timestamp of the latest sample recorded in a Tableau server from one of the APIs?
You can set the incremental's column value in the connector with:
|tableau.incrementalExtractColumn = "id";|
Then this column's value will be your table.incrementValue. See here: