Tableau sends as many data points as there are in the view to R. You only had the calculation itself, so it makes a 1 row table which meant sending 1 rows of data and asking for 5 clusters hence the error message. Below is a screenshot of the working version. I added ID to the view which gives me multiple points (I am assuming 1 for each species in this case) then set the table calc setting to Compute Using > Id.
You also need to update your code to return mod$cluster instead of mod$clusters.
Tableau sends a separate request to R per partition of data. So if you'd like to send all your data at once, you have to put all your dimensions as addressing and nothing in partitioning.
In pre-10, to do this you need to go to Advanced... settings for compute using in Table Calc dialog. Post-10 this means checking boxes next to all dimensions in Table calculation dialog.
I hope this helps.
Also any reason why you're doing k-meas using R. Did you try the built-in clustering?