In your original calc, nothing was aggregated so everything went smoothly. When you added MAX() to the second formula, you aggregated part of the calc, so Tableau wants everything aggregated so it can compute properly.
I have no way to test, but you can try:
IF attr([DOCDATE]) <= TODAY() AND DATEDIFF('month',[DOCDATE], MAX([DOCDATE])) = 0
THEN sum( [SALESAMT])
I could be missing something else, as well.