I do not know of any component of the api code or Tableau Server which is supported on Linux currently. I searched through internal and external docs and found no information on tabserver64.
There is a Tableau Community Idea that you might like to upvote related to Linux hosted Tableau Server.
Sorry I do not have more information
If I recall correctly, the executable is automatically invoked by the API and you do not run it directly (which explains why there is no help). It a background/daemon process.
If you're trying to understand how to use the API, then use the (sparse) accompanying documentation. If you want to understand the internal architecture, there doesn't seem to be any documentation for that.
Thank you for answering. Yes, tableau is indeed a very strange software if you have long running experience with unix.
This binary needs to be installed in your $PATH, and only then you can run the compiled programs with C API.