Tableau Prep can definitely handle >12M rows. One of my workflows generates ~500M rows.
You might be facing the issue probably because of lesser Memory in your machine. From my understanding, Tableau Prep creates temporary tables on the local machine during the flow execution. Note that If your workflow contains data cleansing steps and data manipulation steps,joins, unions etc all those operations will be handled by your local machine. If Tableau prep is installed on the C: Drive, you need to ensure that your C: Drive has enough memory to accommodate the temporary tables.
Hope this helps