I have seen this error in the past when Tableau is connected to an external service.
In the instances reviewed, the error message occurred when an invalid data type is received. Tableau can only import the following data types from an Rserve server:
RDouble, RInteger, RLogical, RCharacter
More information about the errors you can receive when Tableau is connected to an external service are documented in the article below.
I also recommend to have a look to R Implementation Notes in the community document below.
Last but not least the possible workarounds options below may help.
1. Run code in R to verify that the value being returned is what is expected (use attached file for type testing template)
2. Use the SCRIPT function that matches the type of value being returned.
3. See the following page on type conversion in R for information on converting data types: http://www.statmethods.net/management/typeconversion.html
Hope this help