I created a calc called "Duration agg" that is SUM([Duration seconds]) and then created a new duration calc using that new calc instead:
//replace [Duration seconds] with whatever field has the number of seconds in it
//and use a custom number format of 00:00:00:00 (drop the first 0 to get rid of leading 0's for days)
IIF([Duration agg] % 60 == 60,0,[Duration agg] % 60)// seconds
+ IIF(INT([Duration agg]/60) %60 == 60, 0, INT([Duration agg]/60) %60) * 100 //minutes
+ IIF(INT([Duration agg]/3600) % 24 == 0, 0, INT([Duration agg]/3600) % 24) * 10000 //hours
+ INT([Duration agg]/86400) * 1000000 // days
The reason why this works is based on the order of operations. There are three levels of calculation in Tableau - record level, aggregate, and table calc. The duration calc that I'd posted and you used is a record-level calc so it's being computed for each record in the data, and this works just fine in the Individual records view. However when the data is aggregated (such as in a subtotal or grand total) the record level result (that is formated for dd:hh:mm:ss) is being computed and then that result is being aggregated in the subtotal or grand total and returning the nonsensical results.
So instead we aggregate before applying the custom formatting and everything works. Here's a view with the accurate duration agg:
Here's a Tableau Public link: https://public.tableau.com/views/durationformattingagg/Individualrecords?:embed=y&:display_count=yes&:showTabs=y
I'll update my post sometime in the next week to help head off this problem!
That seems to have done it, thanks Jonathan!