8 Replies Latest reply on Dec 20, 2016 3:47 AM by harika appani

    Tableau Server logs filling up C drive--causing work interruption

    John Munoz

      I'm running Tableau Server Version 8.0.0 (Build 8000.13.0319.1225). Yesterday, I received reports saying that people couldn't access the server. We figured out that the problem was that the C drive was filled up. Turns out, Tableau had written over 60GB of log files to:

       

      C:\ProgramData\Tableau\Tableau Server\data\tabsvc\vizqlserver\Logs

       

      I cleared out the files w/o too much trouble and got the server up and running again. I figured I'd have a few weeks to figure out how to implement something like this from happening again. Well, today it happened again. Now there's one file, backgrounder_0_2013_07_26_14_15_56 that's taking up 22GB. We do a lot of extract refreshes and I have a hunch this is causing the problem, but I don't know.

       

      Questions:

      1. Quick fix?
      2. Why is this happening?
      3. Is Tableau going to do something to prevent this from happening, or at least warn Admins when this is about to happen?

      Thanks.

        • 1. Re: Tableau Server logs filling up C drive--causing work interruption
          Tracy Rodgers

          Hi John,

           

          The following KB article should provide some insight on this issue and describe best practices:

           

          http://kb.tableausoftware.com/articles/knowledgebase/troubleshooting-server-backup-and-restores

           

          Hope this helps!

           

          -Tracy

          • 2. Re: Tableau Server logs filling up C drive--causing work interruption
            thierryjakircevic

            Tracy,

             

            We are facing the exact same issue as John. The size of the log files created by Tableau is huge and dataengine will often fail due to a lack of available space on our C drive.

            I do not see how the link you provided help solve the issue. Did I miss something may be?

             

            Also, what about point 3 of John's email?

             

            Please advise as we are running in this issue very often.

             

            Thanks!

            Thierry

            • 3. Re: Tableau Server logs filling up C drive--causing work interruption
              Gerardo Varela

              We have a very small shop but also very limited resources (which I'm finally getting some more at the end of the next month)

               

              Quick Fix Powershell script:

              cd "C:\ProgramData\Tableau\Tableau Server\data\tabsvc\vizqlserver\Logs\"

              Get-Childitem | Foreach-Object {if ($_.LastAccessTime -le (get-date).adddays(-14)) {remove-item -recurse -force $_}};


              Set the number of days to what you need.

               

              Another quick fix to clean up all logs.

              CD C:\Program Files\Tableau\Tableau Server\<VS>\bin"

              tabadmin stop

              tabadmin cleanup

              tabadmin start

              tabadmin cleanup

               

              Save that as a bat script and run on a schedule.

               

              Why is could be happening ? Just a hunch:

              Understanding Disk Space Requirements for Tableau Server | Tableau Software

              Refreshing Extracts:

              If using extracts, consider the space needed by the temp directory during an extract refresh. The temp directory, which is where an extract is stored to during a refresh, may require up to the square of the final file size of the extract. For example, a 12 GB extract may take up 144 GB of disk space to complete the refresh.

               

              Try to stagger the extracts or have the backgrounder services not run in parallel.

               

              You could also point the Tableau's temp folder off  C:\ :

              Changing the Default TEMP Location for Creating Extracts | Tableau Software

               

              From a link right above yours in the admin section of Tableau community:

               

              Log file explodes in size and fills hard drive

               

              And the not so easy fix is to get more disk space

               

              Regards,

               

              Gerardo

              • 4. Re: Tableau Server logs filling up C drive--causing work interruption
                Glauber Ribeiro

                We run a daily backup script that, among other things, does

                 

                tabadmin ziplogs

                tabadmin cleanup

                 

                So we always have the previous day's logs available in case we need to send them to Tableau support.

                 

                We've been running Tableau Server since v5. We learned back then, that Tableau Server was happier if restarted at least daily, so we do that too. I don't know if that's still the case in the newer versions, but it doesn't seem to hurt. It used to be that backup had to be done with the server down, but that's not the case anymore, so we're only down about 10 minutes a day.

                 

                The script flow is, more or less:

                 

                tabadmin stop

                tabadmin ziplogs

                tabadmin cleanup

                tabadmin start

                tabadmin warmup

                tabadmin backup

                • 5. Re: Tableau Server logs filling up C drive--causing work interruption

                  John,

                  Some of the best practices described above in KB and by Glauber Ribeiro are helpful as strategies for ongoing maintenance and management of server.

                   

                  >> Now there's one file, backgrounder_0_2013_07_26_14_15_56 that's taking up 22GB.


                  However, when you see really large (22GB?) files, these are sometimes error_dump files that are created because one of the server processes crashed for one reason or another. It's not clear what is going on in your case based on your description. Highly recommend you consider moving to 9.0 (not as a solution to this problem, but 9.0 has more resource monitoring built into to the system).

                   

                  There are two ways you can proceed, you can file a support case with our team to help triage to see whether these are error dump files due to a process crash or if this is something else. Is it just the backgrounder or some other processes are creating larger files.

                   

                  Alternately, you can approach this problem from the perspective of whether your server processes are starved for resources. If you have an under powered system and you are running under resource stress conditions, that could sometimes lead to process restart by design as some of the server process monitor their health.


                  Last but not least, from a product perspective, in V9 we added resource monitoring to the server processes where processes are smarter to monitor to some threshold levels configured and can try to heal with a restart where possible.


                  We are looking at adding specifically more disk usage monitoring to server, so we can help you be more proactive about these situation. Clearly you can monitor for this today via any infrastructure monitoring solution and take proactive actions like run a cleanup on low disk space, but it would be easier if Tableau would do that automatically.


                  I know I am not solving your specific problem, but my intent is to share a few approaches you could take.


                  Our support team should be able to help triage this better if you open up a case with them.

                   

                  Best,

                  Neelesh Kamkolkar

                  Product Manager

                  Tableau Software

                  • 6. Re: Tableau Server logs filling up C drive--causing work interruption
                    harika appani

                    Hi,

                    Do we have any option that log files can be created on monthly basis rather than daily basis?

                     

                    Regards,

                    Harika.

                    • 7. Re: Tableau Server logs filling up C drive--causing work interruption
                      Russell Christopher

                      No.

                       

                      You should either install Tableau Server on a drive which has enough space to accommodate the size of the logs, or consider creating some sort of a scheduled batch file which executes tabadmin cleanup --restart to cleanup / delete your log files.

                      • 8. Re: Tableau Server logs filling up C drive--causing work interruption
                        harika appani

                        Hi Russell,

                        Thank you for your response,

                        Just curious to know..

                        Are there any effects if  i delete log files?

                        If i delete log file does usage dashboard based on (users,days) on server will show up?

                         

                        Regards,

                        Harika.