Metadata Automation
Jeff Strauss
If you found this post, then wonderful!!!. I hope it helps on your journey to a fully discoverable Tableau Server. If you know somebody that hasn't found it yet, then please pass along.
Before I get started, I want to thank community leaders that inspired my team to go and create the below solution. All of it helped lead us to the path that I will share today.
- Andy Cotgreave for an insightful historical post "TWB Auditor". It's still highly useful for desktop deployments or when you have a collection of workbooks available. What's in your workbooks? The TWB Auditor
- Toby Erkson for enhancing the "TWB Auditor" in order to permit integration with pre-10.x Tableau Server. His creativity, knowledge and community sense never ceases to amaze. What's in your workbooks? The TWB Auditor v2.0
- Mark Wu and Jordan Scott for allowing me to present a webinar last Friday 2/24. If you want to view it on-demand, see this post. Tableau Administrators Virtual User Group: February 24
------------------------------------------------------------------------------------------------------------------------------------------------------
The value goes like so. Analysts create Insightful dashboards and datasources...Publish...Tableau Server has a mass of dashboards (hopefully neatly organized into projects), but now the need arises to peel back the onion, understand collectively the data lineage, and answer questions like:
1. Our DW is replacing X with Y, how are all the dashboards affected?
2. What vizzes have a particular data attribute so that I don't have to start from scratch?
3. What is the underlying calculation and are calculations consistent across workbooks?
4. What is the min and max value for a data dimension?
5. Are there unused fields within published datasources?
Sounds good. So how does this solution work? (open sourced at https://github.com/JeffStrauss18/TableauMetadataExtractor).
(DISCLAIMER: What is on github is AS-IS, there is no guarantee, it will work with 10.1, if you want it to work with earlier versions, it will work with a few small adjustments).
(DISCLAIMER: This is not an internal Tableau developed project. So don't go asking support as it's not officially Tableau supported).
- Create a Metadata Postgres staging instance and a Metadata Postgres prod instance. All the definitions are available. It will hold a copy of core Tableau internal Postgres tables (i.e. workbooks) along with any necessary data enrichments.
- Log into the internal Tableau Postgres as admin and issue a GRANT to pg_largeobject. If you don't want to have to do this in the future, then go upvote idea https://community.tableau.com/ideas/4667
- Create a dblink between the newly created Metadata Postgres staging and your Tableau internal Postgres
- *** Work with the C# Metadata Factory. You will need to modify app.config to point at your metadata instance and then compile it to an .exe and then schedule it at a reasonable frequency based on your deployment. This module is the secret sauce that populates table "fields" and table "xref_workbook_datasource". At a high level, it unzips the XML from pg_largeobject, flattens out the XML, retrieves statistics on cardinality, min, and max values, and does a few data enrichments (i.e. unused fields). The executable will drive the pull from Tableau metadata to the Metadata Postgres staging, will do the parsing, and then it will populate the Metadata Postgres prod.
- Once the data is available within the Metadata Postgres prod schema, then you can create whatever metadata reports that you want. Within github are two examples.
------------------------------------------------------------------------------------------------------------------------------------------------------
Perfection? No, not quite. Or probably more like far from it. But, this can be community driven. If you have ideas, etc. then let's see if we can work together.
Will it work with future iterations of Tableau? Don't know, but we are as of now committed to keep it working as we upgrade. Oh, and I already said that it works with 10.1, it will work with earlier versions with a few small adjustments.
What if there are errors? It takes some setup to get it all functional, but once it's going, it's there. Feel free to respond here or email me if you want at: jstrauss@conversantmedia.com or jeffstrauss18@gmail.com
Cool? If so, thanks. If not, that's ok too.
Want to see a demo? See this post. Tableau Administrators Virtual User Group: February 24