thanks for posting.  we haven't upgraded yet, but we are interested in the outcome of your monitoring / discussion with support.

What was the previous version(10.2.1,10.2.2 or 10.2.3) before the upgrade.I will be upgrading the test environment from 10.2.2 to 10.3.1 this week.

We've been running into this as well.  Upgraded from 10.1.3 to 10.3.0 and have noticed Data Server processes dropping frequently.  Just disabled "Workbook Performance after a Scheduled Refresh"

Hoping this resolves the issue, will post back after a few days with my findings

Ok, I understand that to avoid that we have to turn-off one of new feature of 10.3.*

Do you know if Tableau will fix it?

Not sure that turning off the new feature is a good way to fix it. Also it seems it is nice feature.

I'm replying for Jon as he is on vacation this week.  We still have the ticket open with Tableau Support.  The auto cache warming is a great concept and we'd actually love to use it.  Turning it off fixed the errors and constant data server crashes so we were able to keep the upgrade and carry on with our work.  However, we aren't viewing this as a fix.  Instead, it was a workaround to get us back into a production state.

I'd love to be able to use this new feature.  We did find that it was consistently crashing after 3 specific extracts refreshed.  All three extracts are a bit large with 100+ workbooks created from each of them.  All of the smaller extracts with fewer workbooks had no trouble with auto cache warming.

If anyone else finds more info on this issue, please share. Thank you!!

*note:  Support stated that it could be a resource issue.  However, we are running a 16 core box with 256 GB of RAM.  We're hardly hitting 50% cpu and 15% ram. 

HI Sarah,

Regarding the "may-be-performance-issue",

1) could you please tell how many TS processes

(# of each) are running on your box?

# Backgrounders (4)

# Data Servers (2)

# Data Engines (2)

# VizQL Servers (2)

# Cache Servers (2)

*in () are some good numbers to start with :-)

2) where (the time) and how often the refreshes

for those three (published) extracts in question are scheduled?

Are they finished (and their warmings started) closely in time?

I could imagine a huge performance hit when

(on a behalf of those 100s of connected workbooks)

Backgrounder(s) are starting to send

hundreds+ of queries to Data Engine(s) --

each of them is proxied by Data Server(s) --

in aim to fill the (Redis) Cache with fresh objects.

Yours,

Yuri

Our setup:

# Backgrounders (6 - We realize this is out of the ordinary but it's working quite well.  We tested the cache warming with 4 and 6.  We do realize that at some point in the future we will need a second node.)

# Data Servers (3)

# Data Engines (1)

# VizQL Servers (2)

# Cache Servers (2)

Refreshes Day 1 of issues:

0100 - Refresh Extract Starts

0124 - Refresh Extract Ends

0124 - External Query Cache Warmup Job Starts

0319 - External Query Cache Warmup Job Ends -  Crashes.  This should have timed out at 10800 sec but ran for 93253 sec.  The timeout value is even noted in the log.

The rest of the day we were running extracts with no issue but the majority of Cache Warmup jobs failed. Many of them didn't fail/crash/stop until we rebooted the server the next day.

-Skipping Day 2 because everything was behind due to the Data Servers crashing  - related to the Cache Warming (see comments near the bottom)

Refreshes Day 3 of issues:

0100 - Refresh Extract Starts

0126 - Refresh Extract Ends

0126 - External Query Cache Warmup Job Starts

0917 - External Query Cache Warmup Job Ends - Crashes with an error Workbook cannot be found (we looked up the workbook number and it was there).  It also should have timed out much sooner than it did.  - Data Servers started crashing shortly after this

We run extracts from 0100 - 0900 each day.  We were doing this with no issues with 4 backgrounders prior to the cache warming feature.  Many are extremely fast, seconds and scheduled with enterprise software using tabcmd. We schedule long running extracts with short, quick, extracts so there aren't more than 4 or so running at the same time. Only a few run for more than 15 minutes (we call these the long running extracts) and those are spread apart by a couple of hours due to the nature of when the data is loaded for them. 

We then added 2 backgrounders (6 total) and this did not help with the cache warming.  My assumption is that the cache warming adds a background process for each extract, thus increasing the number of processes needed by double and the time to run is based on how many views use the extract (and how much of the extract).  Add the new data alerts background processes that run and everything starts to step on each other. Also only having 1 Data Engine could have slowed us down and contributed to the data server processes crashing (another guess).  If we could turn External Query Cache Warming on for specific extracts, this may help.  However, the on/off is either per site or per server. I would love to see a formula showing how many Server/Engine/Background processes we would need based on the number of extracts we have and workbooks that are built off of them so we could have External Query Caching working. We have the CPU capacity to add processes. 

I did notice that as soon as the cache warming jobs started failing that it was a downward spiral of failures.  The Data Servers would crash, then suddenly dashboard loading would get really slow, and finally tableau server would only take users to a blank page.  We then would have to restart services and/or reboot the server. 

*Note:  I am no expert and am still learning quite a bit about Tableau Server. Any suggestions are very much appreciated.

Sarah, thank you for reply.

Adding one more Data Engine is recommended.

This may cut the warming times by half (in theory).

Besides, two instances of Data Server per node would be enough, imho --

the bottleneck is usually either before (backgrounder or VizQL Server)

Or after (either Data Engine or Protocol Server / DBMS when Live).

Two Data Servers are mainly for stability (:-) -- as each one

could open up to a thousand connections simultaneously.

And of course, doubling the number of CPU per node would be nice,

ditto moving Backgrounders to their own node (or two may be :-).

Yours,

Yuri

Yuriy Fal we are also running 2 dataservers, and don't see this at all as a bottleneck.  I do have just one question for you.  your comment around each DS can open up to a thousand connections simultaneously.  Where does this number come from?

Hi Jeff,

tabadmin config -o c:\config.txt

Open c:\config.txt and you could find the line:

dataserver.protocolcachesize: 1000

Yours,

Yuri