12 Replies Latest reply on Jul 6, 2017 11:35 AM by Yuriy Fal

    Upgrade from Server 10.2 to 10.3.1 Performance Issues

    Jonathan Schmiel

      We're on a single node server and recently upgraded from 10.2 to 10.3.1.  Resource usage and backgrounder que shot up from "Warming up external query cache on data change" Background tasks.  Our Data Server process also kept going down.  We noticed that   "Warming up external query cache on data change" never ran prior to our update.  After contacting support, they recommended turning this off by going to each site settings and unchecking "Workbook Performance after a Scheduled Refresh".  This is a new feature in 10.3.  We just applied this and are monitoring to see if this helps. 

       

      More info on instructions to turn off this feature here: Configure Workbook Performance after a Scheduled Refresh

       

       

      Has anyone else had this issue after upgrading to 10.3.1?

       

      Thanks!

        • 1. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
          Jeff Strauss

          thanks for posting.  we haven't upgraded yet, but we are interested in the outcome of your monitoring / discussion with support.

          • 2. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
            sreeni.gorthi.0

            What was the previous version(10.2.1,10.2.2 or 10.2.3) before the upgrade.I will be upgrading the test environment from 10.2.2 to 10.3.1 this week.

            • 3. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
              Jonathan Schmiel

              We upgraded from 10.2.0.  I'll have more info on resolution tomorrow too.

              • 4. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                Jason McCann

                We've been running into this as well.  Upgraded from 10.1.3 to 10.3.0 and have noticed Data Server processes dropping frequently.  Just disabled "Workbook Performance after a Scheduled Refresh"

                 

                Hoping this resolves the issue, will post back after a few days with my findings

                • 5. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                  Jonathan Schmiel

                  After going to each site's settings and unchecking "Workbook Performance after a Scheduled Refresh", the performance issues disappeared and we're back to normal.  Hope this helps others having similar issues!

                  4 of 4 people found this helpful
                  • 6. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                    Egor Larin

                    Ok, I understand that to avoid that we have to turn-off one of new feature of 10.3.*

                     

                    Do you know if Tableau will fix it?

                    Not sure that turning off the new feature is a good way to fix it. Also it seems it is nice feature.

                    • 7. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                      Sarah Akers

                      I'm replying for Jon as he is on vacation this week.  We still have the ticket open with Tableau Support.  The auto cache warming is a great concept and we'd actually love to use it.  Turning it off fixed the errors and constant data server crashes so we were able to keep the upgrade and carry on with our work.  However, we aren't viewing this as a fix.  Instead, it was a workaround to get us back into a production state.

                       

                      I'd love to be able to use this new feature.  We did find that it was consistently crashing after 3 specific extracts refreshed.  All three extracts are a bit large with 100+ workbooks created from each of them.  All of the smaller extracts with fewer workbooks had no trouble with auto cache warming.

                       

                      If anyone else finds more info on this issue, please share. Thank you!!

                       

                      *note:  Support stated that it could be a resource issue.  However, we are running a 16 core box with 256 GB of RAM.  We're hardly hitting 50% cpu and 15% ram. 

                      • 8. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                        Yuriy Fal

                        HI Sarah,

                         

                        Regarding the "may-be-performance-issue",

                         

                        1) could you please tell how many TS processes

                        (# of each) are running on your box?

                         

                        # Backgrounders (4)

                        # Data Servers (2)

                        # Data Engines (2)

                         

                        # VizQL Servers (2)

                        # Cache Servers (2)

                         

                        *in () are some good numbers to start with :-)

                         

                        2) where (the time) and how often the refreshes

                        for those three (published) extracts in question are scheduled?

                        Are they finished (and their warmings started) closely in time?

                         

                        I could imagine a huge performance hit when

                        (on a behalf of those 100s of connected workbooks)

                        Backgrounder(s) are starting to send

                        hundreds+ of queries to Data Engine(s) --

                        each of them is proxied by Data Server(s) --

                        in aim to fill the (Redis) Cache with fresh objects.

                         

                         

                        Yours,

                        Yuri

                        1 of 1 people found this helpful
                        • 9. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                          Sarah Akers

                          Our setup:

                          # Backgrounders (6 - We realize this is out of the ordinary but it's working quite well.  We tested the cache warming with 4 and 6.  We do realize that at some point in the future we will need a second node.)

                          # Data Servers (3)

                          # Data Engines (1)

                          # VizQL Servers (2)

                          # Cache Servers (2)

                           

                          Refreshes Day 1 of issues:

                          0100 - Refresh Extract Starts

                          0124 - Refresh Extract Ends

                          0124 - External Query Cache Warmup Job Starts

                          0319 - External Query Cache Warmup Job Ends -  Crashes.  This should have timed out at 10800 sec but ran for 93253 sec.  The timeout value is even noted in the log.

                          The rest of the day we were running extracts with no issue but the majority of Cache Warmup jobs failed. Many of them didn't fail/crash/stop until we rebooted the server the next day.

                           

                          -Skipping Day 2 because everything was behind due to the Data Servers crashing  - related to the Cache Warming (see comments near the bottom)

                           

                          Refreshes Day 3 of issues:

                          0100 - Refresh Extract Starts

                          0126 - Refresh Extract Ends

                          0126 - External Query Cache Warmup Job Starts

                          0917 - External Query Cache Warmup Job Ends - Crashes with an error Workbook cannot be found (we looked up the workbook number and it was there).  It also should have timed out much sooner than it did.  - Data Servers started crashing shortly after this

                           

                          We run extracts from 0100 - 0900 each day.  We were doing this with no issues with 4 backgrounders prior to the cache warming feature.  Many are extremely fast, seconds and scheduled with enterprise software using tabcmd. We schedule long running extracts with short, quick, extracts so there aren't more than 4 or so running at the same time. Only a few run for more than 15 minutes (we call these the long running extracts) and those are spread apart by a couple of hours due to the nature of when the data is loaded for them. 

                           

                          We then added 2 backgrounders (6 total) and this did not help with the cache warming.  My assumption is that the cache warming adds a background process for each extract, thus increasing the number of processes needed by double and the time to run is based on how many views use the extract (and how much of the extract).  Add the new data alerts background processes that run and everything starts to step on each other. Also only having 1 Data Engine could have slowed us down and contributed to the data server processes crashing (another guess).  If we could turn External Query Cache Warming on for specific extracts, this may help.  However, the on/off is either per site or per server. I would love to see a formula showing how many Server/Engine/Background processes we would need based on the number of extracts we have and workbooks that are built off of them so we could have External Query Caching working. We have the CPU capacity to add processes. 

                           

                          I did notice that as soon as the cache warming jobs started failing that it was a downward spiral of failures.  The Data Servers would crash, then suddenly dashboard loading would get really slow, and finally tableau server would only take users to a blank page.  We then would have to restart services and/or reboot the server. 

                           

                          *Note:  I am no expert and am still learning quite a bit about Tableau Server. Any suggestions are very much appreciated.

                          • 10. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                            Yuriy Fal

                            Sarah, thank you for reply.

                             

                            Adding one more Data Engine is recommended.

                            This may cut the warming times by half (in theory).

                             

                            Besides, two instances of Data Server per node would be enough, imho --

                            the bottleneck is usually either before (backgrounder or VizQL Server)

                            Or after (either Data Engine or Protocol Server / DBMS when Live).

                            Two Data Servers are mainly for stability (:-) -- as each one

                            could open up to a thousand connections simultaneously.

                             

                            And of course, doubling the number of CPU per node would be nice,

                            ditto moving Backgrounders to their own node (or two may be :-).

                             

                            Yours,

                            Yuri

                            1 of 1 people found this helpful
                            • 11. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                              Jeff Strauss

                              Yuriy Fal we are also running 2 dataservers, and don't see this at all as a bottleneck.  I do have just one question for you.  your comment around each DS can open up to a thousand connections simultaneously.  Where does this number come from?

                              • 12. Re: Upgrade from Server 10.2 to 10.3.1 Performance Issues
                                Yuriy Fal

                                Hi Jeff,

                                 

                                tabadmin config -o c:\config.txt

                                Open c:\config.txt and you could find the line:

                                dataserver.protocolcachesize: 1000

                                 

                                Yours,

                                Yuri

                                1 of 1 people found this helpful