1 Reply Latest reply on Feb 22, 2017 4:19 PM by diego.medrano

    Dynamic extracts from Bigquery Date partitions?

    Jarle Sandnes

      Hey, I have an idea about what I want to do and think it should be possible but trying to wrap my head around how to implement it. I have a Tablaeu dashboard which displays data from a Date partitioned table in Bigquery, and I want to optimize performance while minimizing cost and load in bigquery. The table itself is about 300GB/A few billion rows.

       

       

      I imagine the optimal way to handle this is doing one extract representing all data, and then one extract from the newest partition containing all new data, refresh that every hour or so, and every 24 hours appending that extract to the main extract? Seems like a common use case but haven't found any specific implementations like it.

       

      Mainly unsure about how extract works, do I just create an extract from all relevant columns of the database and tablaeu is smart about it in terms of what metrics are needed for the dashboard? Or do I need to specify every type of specific aggregate I might want to get best performance? Also not sure how to only refresh the extract of newest date partition and append but guess I might need 2-3 data sources

       

      Any input is much appreciated.