7 Replies Latest reply on Feb 17, 2017 1:19 AM by Benjamin Söllner

    Admins, Do You Let Publishers Publish Directly to PROD?

    Benjamin Söllner

      Hi all!

       

      I am a platform architect and will be handling over my Tableau Installation pretty soon from one operations team & lead administrator to another one.

       

      We have a fairly small Installation with only a few critical projects, most reports deployed to our server are not business critical. Still, we have a full-featured DEV, INT and PROD installation because we want to roll out platform upgrades in a smooth way through all three environments providing ample time for testing.

       

      Right now, all users can deploy their workbooks to whichever environment they want. They can follow their project-specific staging patterns or deploy directly to PROD. My new admin, however, suggests that we "lock down" INT & PROD and work with ticket systems, screening individual workbooks on INT before migrating them to PROD. I am not a fan of this proposal: not only would it require large efforts, additional infrastructure and staff, it would slow down iterating on workbooks with live data a lot (no "analysis at the speed of thought" anymore). I think it would disappoint a lot of our users.

       

      I would be interested in hearing a few "third" opinions from as many of you as possible. Here are some guiding questions.

       

      1. Did you open all environments up to your publishers, including PROD or do you have custom/scripted deployment process?
      2. Was there an event in your history with Tableau that made you reconsider how "open" you leave platforms for this kind of "self-service"?
      3. Did you establish checklists for staging between DEV-INT-PROD? Any things that should definitely be on those checklists? Who enforces them?
      4. What is the worst that can happen with a faulty workbook? Can it poison a shared Tableau server enough to make it practically unusable (given workbooks live in different sites)?
      5. Does monitoring, set up in a right way, provide enough resilience that I can just "shut down" a faulty workbook (taking a rather "optimistic" than "pessimistic" approach to published workbooks)?

       

      Looking forward to your stories. Thanks!

       

      Best wishes,

      Benjamin

        • 1. Re: Admins, Do You Let Publishers Publish Directly to PROD?

          Hey Benjamin,

           

          If I recall correctly, a few years ago I saw David Darden gave a presentation on these sorts of questions faced at Big Fish Games. I wonder if David would be willing to chime in with their experience.

           

          I've seen various forms of content control across different companies. My personal preference has been to have a strict set of guidelines that a viz must meet in order to be published to PROD. PROD is open but all workbooks published are reviewed by an admin who then either says the viz can stay, or moves it to a different instance and gives reasoning why.

           

          Matt Coles thoughts from an admin's end?

           

          -Diego

          • 2. Re: Admins, Do You Let Publishers Publish Directly to PROD?
            Patrick A Van Der Hyde

            At least for the workbooks on some of the teams here at Tableau, we screen new data sources a bit more carefully and do not allow those to go straight to prod but we allow workbooks built off of existing data sources (single source of the truth) to be published.  We still suggest peer reviewing though. 

             

            Patrick 

            • 3. Re: Admins, Do You Let Publishers Publish Directly to PROD?
              Russell Christopher

              Hey Benjamin -

               

              I think you're going to get some good "nuts and bolts" replies here, so I'm going to attempt to approach this from a different direction.

               

              Start with the question of why your company bought Tableau... It's not unusual "for the reason" to get lost in the shuffle between purchase and implementation. Most Administrators want to administer- so they're going to lock that sucker down and come up with all sorts of cool processes to layer on top of the server. But are they necessary? For example, if "the reason" for purchase is to enable an analytical culture in the org, "process" is the exact opposite of what you want.

               

              Do you even NEED to Dev > Int > Prod? This process is a little old school and typically only necessary for the top x % of "reports" that are mission critical to the business. Attempting to curate 100% of the stuff on your server is going to take lots of effort and will (absolutely, positively) stifle creativity, collaboration, etc. If too much friction is added "the process" of getting stuff out on Tableau, users simply won't bother - they'll sneaker-net their stuff around or hide/share it on network drives....and it's "Excel ****" all over again.

               

              I think you get where I'm going - have a site or project on the server where only "blessed" vizzes live - but even they may not really need to go through a formal migration process. Then, have several "wild, wild west" areas which are pretty much sandboxes..sandboxes which are completely cleaned out now and then since your folks generally won't clean up after themselves 100% of the time. Also have a few "semi-curated" areas where semi-permanent / important stuff.

               

              You can monitor viz execution time using Tableau's built in admin vizzes and come u with your own max run time rules...if any vizzes take longer than X to run, the owner gets a call from your team. (Keep in mind that you'll also need to have resources you can send these folks about HOW to improve their work...hopefully there are a couple of "champs" in the business who are willing / able to assist).

               

              There's also a great tool called LogShark which will parse your logs for you and create some very in-depth reports. With a little bit of extra work, you can even do things like track the fact that Dashboard "Foo" (which was executed 10 times last week) contains X queries, and that query #2 of X executes in 30 seconds, while queries #1, #3, #4 only take 2 seconds....This might be overkill for you, though.

               

              So at the highest level, I'd suggest you treat your Tableau Server like an English Garden (vs. French) - a little wild, a little rough around the edges, but ultimately much more interesting and enjoyable.

              • 4. Re: Admins, Do You Let Publishers Publish Directly to PROD?
                Toby Erkson

                Hey Ben! 

                Benjamin Söllner wrote:

                 

                ...

                Right now, all users can deploy their workbooks to whichever environment they want. They can follow their project-specific staging patterns or deploy directly to PROD. My new admin, however, suggests that we "lock down" INT & PROD and work with ticket systems, screening individual workbooks on INT before migrating them to PROD. I am not a fan of this proposal: not only would it require large efforts, additional infrastructure and staff, it would slow down iterating on workbooks with live data a lot (no "analysis at the speed of thought" anymore). I think it would disappoint a lot of our users.

                 

                -- You are 100% correct in your assessment   If the proposed process by the new admin is what that person desires then tell them to go see the Cognos team, they would love to have him/her!

                Benjamin Söllner wrote:

                I would be interested in hearing a few "third" opinions from as many of you as possible. Here are some guiding questions.

                 

                1. Did you open all environments up to your publishers, including PROD or do you have custom/scripted deployment process?
                2. Was there an event in your history with Tableau that made you reconsider how "open" you leave platforms for this kind of "self-service"?
                3. Did you establish checklists for staging between DEV-INT-PROD? Any things that should definitely be on those checklists? Who enforces them?
                4. What is the worst that can happen with a faulty workbook? Can it poison a shared Tableau server enough to make it practically unusable (given workbooks live in different sites)?
                5. Does monitoring, set up in a right way, provide enough resilience that I can just "shut down" a faulty workbook (taking a rather "optimistic" than "pessimistic" approach to published workbooks)?

                ...

                1.  Yes.  Why?  Keeping with the self-service paradigm.  One less IT bottleneck.  Less maintenance during Tableau upgrades, less things that will 'break'.

                2.  Not once.

                3.  No.  We only use QA and PROD.  Authors test and UAT (User Acceptance Testing) on QA.  Once it looks good the author publishes to PROD and deletes it from QA.

                Note:  Our QA Server has only one schedule end users can use -- since an author can manually execute an extract there is no need to put it on a schedule that is the same as one on PROD.  This dramatically reduces process consumption, allowing for a slightly smaller server resource if necessary (smaller companies that can't afford exact hardware matched environments.  It also keeps them from "oops, I forgot!" moments and thus not using QA as a PROD Server   My lesson learned.

                Note:  I use the third allowable server, TEST Server, for beta testing and very special testing/trouble-shooting for some reports and once done any users & workbooks are quickly deleted by me.  This server meets the minimum requirements for a Tableau Server installation as a testing environment and could not be used for production (only has 4 cores).  Again, saving money.

                4.  By setting backgrounder limits the admin can keep things from getting too out of control.  I have a server setup script run after each time I upgrade the Servers and I have this in it:

                REM Longest allowable time for completing an extract refresh, in seconds (10800 seconds = 3 hours)
                tabadmin set backgrounder.querylimit 10800
                REM Seconds beyond the query limit before a task is canceled, in seconds (300 seconds = 5 minutes)
                tabadmin set backgrounder.extra_timeout_in_seconds 300
                REM Tasks to be canceled if they run longer than they should
                tabadmin set backgrounder.timeout_tasks refresh_extracts,increment_extracts,subscription_notify,single_subscription_notify
                

                Thus, if an extract is still running past the 3 hour mark it gets killed by the Tableau Server!  This appears to be working well as I've yet to get any complaints.  My logic is that if an extract takes longer than 3 hours then something is not right, plus it's hogging resources that could be used by others.

                In my experience I don't know of a situation where a bad workbook would bring a Server to its knees.

                5.  Yes.  There are various ways to monitor things and it really depends on the situation, thus it will vary between companies and even regions/departments within a company, and they can change over time.  If there is a peer review or other workbook checking method then such a process will likely reduce such faulty workbooks.

                One thing about "bad" workbooks is that if they are too slow then the majority of the time the end user will complain and that goes straight back to the author to have them improve it.  The performance recorder is a great tool for report authors and server admins.  Time and again when authors complain about my Server as being slow and to "fix it" it turns out it's their workbook that is at fault.

                 

                Russell Christopher wrote:

                 

                ... Most Administrators want to administer- so they're going to lock that sucker down and come up with all sorts of cool processes to layer on top of the server. But are they necessary? For example, if "the reason" for purchase is to enable an analytical culture in the org, "process" is the exact opposite of what you want...

                I know you said "most" but I am not one of them   I keep busy enough and locking something down that isn't meant to be locked down is unnecessary additional work, so follow Russell's advice.

                Russell Christopher wrote:

                 

                ...Do you even NEED to Dev > Int > Prod? This process is a little old school and typically only necessary for the top x % of "reports" that are mission critical to the business. Attempting to curate 100% of the stuff on your server is going to take lots of effort and will (absolutely, positively) stifle creativity, collaboration, etc. If too much friction is added "the process" of getting stuff out on Tableau, users simply won't bother - they'll sneaker-net their stuff around or hide/share it on network drives....and it's "Excel ****" all over again.

                Yup!  Again, Ben, welcome to our Cognos environment.  One cannot think of Tableau as "just another BI tool"!  Getting old school IT personnel to understand and accept this is not easy   You have to be firm, look them in the eye, and tell them "Trust me, I know what I'm doing, I'm a professional"

                Image result for freightliner

                • 5. Re: Admins, Do You Let Publishers Publish Directly to PROD?
                  Justin D'Cruze
                  1. Did you open all environments up to your publishers, including PROD or do you have custom/scripted deployment process? Sandpit and QA sites are open to varying degrees, Prod is locked down (except certain Sandpit projects for web authoring off published data sources)
                  2. Was there an event in your history with Tableau that made you reconsider how "open" you leave platforms for this kind of "self-service"? Yep we re-visit this quite frequently
                  3. Did you establish checklists for staging between DEV-INT-PROD? Any things that should definitely be on those checklists? Who enforces them? No Dev, instead of separate Server environments we use different sites for QA/Prod. Checklist generally revolve around performance, extract size/refresh time, etc
                  4. What is the worst that can happen with a faulty workbook? Can it poison a shared Tableau server enough to make it practically unusable (given workbooks live in different sites)? There's a lot of variables, but generally not. Having said that....I'm currently investigation a workbook which returns in 25 secs as a packaged workbook, but then times out if running off the same published data sources and causes one data engine process to just sit on >50% CPU for a good 15 mins. Not great for server performance
                  5. Does monitoring, set up in a right way, provide enough resilience that I can just "shut down" a faulty workbook (taking a rather "optimistic" than "pessimistic" approach to published workbooks)? I'd say Tableau's audit tables combined with Tabmon/Performance monitor data is usually enough to pinpoint most issues
                  • 6. Re: Admins, Do You Let Publishers Publish Directly to PROD?
                    jegan.sivaraj

                    How big of an environment do you have? Ours is a 16 core single server installation. There are about 40 desktop users and 300 daily users on the server.

                     

                    1. Did you open all environments up to your publishers, including PROD or do you have custom/scripted deployment process?

                              Prod is open to all publishers. Each dept has a project (with a leader) and they are responsible for the content. Enterprise data sources are vetted by our team but other dept focused extracts are their responsibility.

                    1. Was there an event in your history with Tableau that made you reconsider how "open" you leave platforms for this kind of "self-service"?

                              No.

                    1. Did you establish checklists for staging between DEV-INT-PROD? Any things that should definitely be on those checklists? Who enforces them?

                              We have one server environment. There is a QA/Prod projects for each dept. They are responsible for testing and migration of the workbooks.

                    1. What is the worst that can happen with a faulty workbook? Can it poison a shared Tableau server enough to make it practically unusable (given workbooks live in different sites)?

                              I suppose it could eat up a lot of CPU? I'm not sure. We have never had that issue.

                    1. Does monitoring, set up in a right way, provide enough resilience that I can just "shut down" a faulty workbook (taking a rather "optimistic" than "pessimistic" approach to published workbooks)?

                              Performance monitoring workbooks are available for all publishers. They pretty much self regulate the environment. Of course we provide a nudge now and then but so far it has been ok.

                    • 7. Re: Admins, Do You Let Publishers Publish Directly to PROD?
                      Benjamin Söllner

                      Thank you all for your wonderful responses, it especially delights me to see my distant collegue from the US, Toby Erkson, post here.

                       

                      As to the question of Jegan, we have a 8-core installation with 3 environments (DEV-INT-PROD) and want to use it for 500 publishers max.

                       

                      This discussion was a very good sanity check on how I thought things ought to be. We will stay with a self-service model now.

                       

                      I'm not sure if David Darden or Matt Coles will still late-join the discussion, but if not, that has certainly already been very insightful!

                       

                      Thanks again!