Our organization has 20+ corporate extracts that run every morning and are triggered to refresh based on dependent database processes completing. Although we roll the data up to the month level, most have fairly complex custom SQL behind them and they all report on 13 months worth of data so it takes awhile to do full refeshes.
I've started exploring ways of speeding up the process, and one of those is creating two versions of the extract - one for the previous 12 months (12MO) and one for the current month (MTD), and then only refreshing the MTD daily and appending that to the static previous 12MO extract. I've started building batch files that use TabCmd statements to kick off a set of test extracts, but am running into issues as the extracts can (and often do) run concurrently and one file's 'tabcmd logout' command is affecting the other batch files (all of the batch files are running on a trusted server with TabCmd installed on it).
At a high level, the following steps are being performed within multiple batch files sitting on a Staging site on the server;
- Log into Tableau Server and kick off the MTD extract in synchronous mode (I want to wait for it to finish before proceeding)
- When MTD extract is complete, get/download a copy of the 12MO extract and the refreshed MTD extract to a temp directory on the trusted server
- Publish the 12MO extract back to the staging site/server with the production name, overwriting the file from yesterday
- Append the MTD file to the 12MO file that was just published
- Get/download the newly-created production extract, and republish it to a different Production site on the server
- Issue a tabcmd logout command
Stand-alone the processes seem to run fine and complete successfully, but when I try to run multiple batch files concurrently I run into issues when one completes while others are still running (it is basically logging out the other TabCmd processes). I know that since I am running under a single user on the trusted server when I am logging out of one process I am removing the token that I got from server, so to tried to work around this by adding an explicit tabcmd login statement to the individual scripts within a batch file. That seems to work better, but with multiple batch files running I am still seeing logout commands step on other batch files.
Wondering if I am trying to do too much with TabCmd? If this a process that is better done through the API, or am I just missing a step somewhere? Not issuing a logout command would solve the problem, but I'm guessing that is not best practice.