3 Replies Latest reply on Nov 27, 2017 9:13 AM by lenaic.ridinger

    Tableau SDK: TDE file generation

    siva.puvvula

      we are building spring batch based app to generate TDE files.

      • Connects to database and retrieve data
      • Write to a TDE file using SDK
      • This runs once a day

       

      Here are the problems we are running into to

      • After few runs of the programs it suddenly starts failing with Caused by: com.tableausoftware.TableauException: Cannot allocate memory on below line (it is trying to create extract file)

           extract = new Extract(file);

      • Before this line of code we did the file FileUtils.deleteQuietly(new File(file));

       

      Application is running on 64bit linux machine with 12GB Heap.

      Currently no TAB_SDK_TMPDIR set, so Tableau extract is using /tmp which has 2GB volume.

       

      • Why does program starts failing with above error?
      • Also, I have seem temp file left over in the directory where I am creating extracts (xxxxx.tde.temp). Why are these files left behind?

       

      I am closing extract extract.close();

        • 1. Re: Tableau SDK: TDE file generation

          Hi siva.puvvula,

           

          Are you using blended databases?

          If yes, could you try to create a single connection that includes all the columns from multiple tables by defining all joins between tables as part of the data connection?

           

          When Tableau Server is installed in an environment with dynamically allocated memory, changes in RAM allocation might result in insufficient memory allocation for Tableau Server.

           

          Resolution

          1. Create a full backup of Tableau Server. For more information, see Back Up Tableau Data in Tableau Help.
          2. Work with your IT team to reconfigure the virtual environment to use static RAM instead of dynamically allocated memory. Verify that the RAM allocation meets the Tableau Server Technical Specifications.
          3. On the Windows Start menu, navigate to Control Panel > Programs and Features, click Tableau Server, and then click Uninstall.
          4. Delete the Tableau Server folder from the following location:
            • On a 64-bit machine: C:\Program Files (x86)\Tableau\Tableau Server AND C:\ProgramData\Tableau\Tableau Server
          5. Restore the backup with the "--no-config" option. For more information, see Restore from a Backup in Tableau Help.

           

           

          ----------

          Lénaïc RIÉDINGER, Global Community Engineer Tableau

          Tableau Community Forums | Knowledge Base

          If you see a Helpful or Correct response, please mark it thanks to the buttons below the targeted post!

          • 2. Re: Tableau SDK: TDE file generation
            siva.puvvula

            Thanks for the input.

             

            I made some progress with debugging.

            This error is so vague, didn't know where to start (no log in DataExtract.log).

             

            After started monitoring every aspect of machines resources, In my case it happened to be when there isn't enough RAM when tdeserver starts.

            How much RAM is enough for tdeserver to start? now sure.. any one know how much should be enough?

             

            This machine has 16GB and running only one java process to generate these tde extracts using SDK.

            So, I have reduced Heap allocation for the the Java process to 5GB, estimating java native will take 3GB more; this process will use 8GB. So there will always will be at least 8GB left for tdeserver and Extract API to use (assuming no other resource hungry process is running on the box).

             

            For now I went ran couple of iterations with expected load and system handled well. But if I have to handle more, I would start with 32GB RAM and tune system to handle the same.

             

            After generating tde files, we proceeded to used REST API to publish datasources successfully.

             

            Use Case: Able to generate TDE files with 3Million records in less than a minute and publish using API's.

            • 3. Re: Tableau SDK: TDE file generation

              Excellent, thank you so much for sharing the resolution with us