Marco Bakker (Member) asked a question.
In most cases, I simply process all the available data in a flow. I don't use the "Append" function based on some unique field.
As the sources grow, I decided to use the Append function in Prep as the sources contain a unique Event_ID (A counter), that should allow for incremental updating:
With the following settings:
The initial run (no database yet) is about 11 minutes. The second run about 3 minutes. So I thought it just worked, but NO. Prep simply appends all the sources again to the output database.
A third run does the same.
In some cases Prep does something else: it selects a random month and outputs only THAT data to the output database.
So either I do something wrong, or Prep is just not working.
For now, I will fall back on filtering on File_Paths. Prep does read all the sources, but only processes the new files based on a join. That seems to work at least.