I am entirely new and downloaded Tableau Public to get a feel.
Am I right in thinking that anything published via Tableau Public is wide open for complete public access?
Or is there a way to keep this data somewhat private?
Nope, it's public. If you don't post the link anywhere it's going to be harder to find, but security through obfuscation is never a sound plan. Consider anything on there wide open, especially as you can get to the raw data as well.