Unfortunately, I am not very familiar with these situations, but I wanted to get in a ping Russell Christopher
Anytime I have struggles with Server related issues, I find his blog to be absolutely amazing!
I also suggest posting this in the Server Admin Community. That forum has less activity, and more dedicated Tableau Server gurus on it to help out.
Best of luck!
Which version of 9.X are you running. ?
Are there any activity going on while executing extracts .Monitoring ram usage and memory did it give any insights on the server ?
I will post it on Server Admin community and tableau love.
When creating extracts (especially big ones) , lots of stuff gets landed to the disk. IT is notoriously stingy on disk when they provision a VM - unless one specifically asks for high throughput (which is what Tableau needs), you tend to get "shared dregs"...This could substantially slow your extract prep down. Check to see what sort of disk throughput you have.
There are really no changes to 9 that will substantially impact the speed at which extracts are created, so I doubt that's it.
The easiest way for you to prove this to yourself is to simply install 8.2 desktop on a machine and extract...then replace with 9.03 and try again. Yeah, I know you're running server (and you could do the same test with Server, too) , but the same stuff is essentially happening under the covers. Heck, your problem could be a simple as networking -- since you're on VMs, your virtual boxes could likely be on a different subnet than Server used to be...and now your requests may be getting bounced around a lot more on the network than they were before.
If there were my problem to solve, I'd rule Tableau out first with the test above. Once you do so, you'll want to use 3rd party tools to benchmark CPU, IO, etc. on the "old" and "new" boxes to see what's different. You might need to use wireshark or other tools to understand how network requests are being routed, too.
Thanks for the reply.
I think we did some tests using 8.2 vs 9, but not back to back to provide comparison, so that makes sense and I will run 2 extracts using desktops.
On network layer, we did run some diagnostics to check the throughput and both servers pumps more or less same.
I haven't compared disk I/O, on physical server, storage is local and on VM it comes from SAN. As it comes from SAN, we tried using SSDs in VM environment.
There is one more variable in the equation.
8.2 was installed on one server with 8 cores and 9 is installed on 2 servers 4 cores each. Worker node is configured with 4 backgrounders. 2 data servers and 1 data engine on Primary. (See image below)
So another reason or followup question could be, if you have 4 backgrounder processes on 8 core machine vs 4 backgrounders on 4 core machine, do they behave differently, means during morning hours, we have to run multiple extracts and if you have 4 backgrounders on 4 core machine, processing power might not be enough? I do see CPU maxing out on working when this particular extract runs.
in nearly all scenarios, splitting a single 8 core is bad. While process isolation can be a good thing, very few are happy with the results of you end up with when there are less than 8 cores to handle either VizQL or Engine/Backgrounder work...you end up with a distributed installation that doesn't do anything especially well...
Sorry to be the one to say this, but just keep it simple. stick everything back on an 8 core vm and see how things go...obviously, you'll want less backgrounders in that scenario, as well. Probably no more than 2.
We decided to distribute it, because while these big extracts would run, we had situations, when desktop users won't be able to connect to server deployed data source, you can imagine what would happen with server users. With distributing processes, most of the time (over 95%) user complains were gone.
I was planning to have one 8 core server, but planning to have 4 backgrounders, as I have seen in past 2 backgrounders not enough to finish all extracts.
I will update after we try these options, reconfiguration is supposed to happen tonight.
Also at this time, not considering to buy any additional core licenses.
Thanks for your help.
Understood - you needed to isolate in order to protect your interactive users. But, you have to realize (and I think you do) that you're attempting to do the same amount of "extract refreshing work" that you used to with half the resources.
If I take in half the calories I used to, I'll start dropping pounds fast - and probably some muscle too. Your 4-core box just can't do as much as the eight core in regards to this processor-intensive work. You simply have less raw horsepower to bring to bear on the task.
Have you read the topics in help on the # of backgrounders to use based on what other services are on the box? You generally do Cores/4 when have "other services" on the machine. When it's an extract-only box, Cores/2 or Cores/4 is generally appropriate...If you have lots of small extracts you might be able to get away with "Cores":
(Skip to about 50-70% of the way down) You're being pretty aggressive by putting as many backgrounders on the machine as there are cores - especially since we already know that your extract building activities are not trivial. You'll also notice in the topic above there is no mention / guidance of how many backgrounders to put on a split 8 core - because we know that scenario generally ends in tears
I suspect you've thought over much of this already and you're in the best place you can be without more hardware to throw at the problem. FYI, you're not alone in this - adoption of Tableau can be viral at companies, and that 8 core you thought would last you a couple of years can get over-utilized FAST.
So in other words you are telling, although backgrounders are single threaded processes, can still run on more than one core? Which also means, in 8.2 setup on 1 physical scenario, 4 backgrounders were using all 8 cores when needed? I don't see that from CPU utilization chart below.
On the other hand when we went live on 4 backgrounders on VM with tableau 9, I see this on worker node where we have backgrounder. See how many times we hit 100% CPU
It's more dramatic when you look at individual day
Thanks for all valuable information.
1 of 1 people found this helpful
The backgrounders are indeed single-threaded, but they don't do the actual work - They're just a scheduler. If you use a tool like SysInternals Process Explorer, you'll see that all the Backgrounder does is launch a copy of tdeserver64.exe which does the extract generation. tdeserver64 is multi-threaded and will eat up as much CPU (on as many cores) as you can give it.
Don't know why you're not seeing this in 8.X, but I'd assume it's because the 8.x machine has twice the processing power as your VM.