So I found, at least, a partial solution. When I am relating the data sources with the scaffold I need to relate them on the level I want to view them. For example, if I want to look at the HOUR level data I need to relate the two Date fields at the HOUR level. This let me see the data I wanted, but it limits me from being able to drill up and down through the different levels.
Anyone know if this is normal? or is it particular to date-time data.