Without having access to the data, I couldn't dig into this.
I have a sneaking suspicion is has to do with the specificity of the underlying granularity of the VizQL query (Tableau's back-end language.)
I haven't encountered this issue, but I rarely do my joins in Tableau for performance reasons.
Can you upload your sheet to SQL server, join there, build a view, and connect Tableau to that view?
Agree with Galen, its hard guess without looking at it.
But, if I were to guess without looking at it, its possible those 4 records are caught in difference in way date/time filter is applied, as you "Specify" a certain date.
Does your Date have time field attached to it? Are you truly looking from 12 am?
Can you try Custom SQL and see how many records you get? vs SQL.