3 Replies Latest reply on Nov 15, 2014 5:48 PM by Matthew Schnars

    Suggest a workflow for appending an extract on a monthly basis?

    James Keuning

      Here is the scenario - I receive log data on a monthly basis. The fields each month are the same, I just append the new data to the existing data. I would like to develop the best workflow for getting this done. Here are my issues:

       

      1. The data consists of a few hundred thousand records. About 20 fields of types text, number, and date.
      2. As I understand things, you cannot delete data from an extract. You need to delete all of the data in the extract and import the corrected data.
      3. The data arrives in a CSV
      4. I need to make certain modifications to the data before it is ready to append.
      5. Currently I make the modifications in Excel; I can probably figure out how to do them in Tableau by creating certain calculated fields.
      6. I can also use Access or SQL to make the modifications.
      7. Currently I use point Tableau to the Excel file and extract the data. Each month I append the new data to the Excel file and I refresh the extract.
      8. I could point Tableau to the new data and append the data to the extract. I have had some weird issues with that but I will test and figure it out if this is the way to go.
      9. I could use Access as the source for Tableau to connect to.
      10. I could use SQL as the source for Tableau to connect to.
      11. I am hesitant to simply append the new data to Tableau because it does happen that we get bad data sometimes and we are provided a complete replacement for a month. As I understand things, I will not be able to remove the bad data from the extract. It really seems like I need an independent source of the complete data set that I can make edits to and point Tableau to that source.

       

      Has anyone encountered these decisions? What did you do?

       

      My next question is going to be: Assume I decide to use Access instead of Excel, how do I extract from the Access table and get all of my current calculations, parameters, and visualizations to respect the new data source?