Seems unusual and most likely unrelated to Tableau. To prove that, use a query tool external to Tableau to create your custom SQL - I think it's called Workbook for MySQL.
If it's not working in the query tool then it won't work in Tableau either.
Some additional hints;
> Reduce the scope of the query and see if it works. Run SELECT TOP 100 a.col1, a.col2 ....... to only get 100 rows. Does that work?
> Do you have a DBA on site? Given that you aren't overly familiar with SQL, I think it's a bad idea to be optimizing things using custom SQL in Tableau. I personally think Custom SQL in Tableau is just a bad practice when collaborating with user users. I'd much prefer to get a DBA who knows what they are doing to create a view in your database which makes the joins correctly.
> You may need to optimize this query by adding indexes to tables etc. This is DBA sort of stuff, you're going to get over your head quickly here!