12 Replies Latest reply on Mar 26, 2015 2:22 PM by Anton Grobman

    Data Extract API erroring and then quitting

    Anton Grobman

      I wrote a Java program that uses the Data Extract API to create a TDE from a CSV file. When I ran my code the very first time against the csv file, I was able to pull 300K+ rows out and export a tde file to my computer. Since then it has never worked again. I had made some changes to my code after it worked successfully, but when I found that it no longer worked I reverted back to the original working code using Ctrl+Z. Still no luck. I put back my changes and have not been able to get a tde to output. I have been able to return a partial result back, of the first 1000 rows, as a tde, so I know my code changes work and the tde API is working. Where the problem lies is when I try to run the code on the full result (300K+), it always gives me an error when trying to add a row to the table but only once, on a row that is a multiple or 1001 (2002, 3003, 6006, 9009). The error is “data engine error” error code 204. But the code then continues going through the rest of the rows up until a certain point, when it stops printing anything out, keeps running, but never stops and nothing gets exported and no error is thrown. Depending on where the first error message happens the second issue always happens on the same row (3003 and 8866, 2002 and 5005). I have looked at those rows and found nothing in the data, I have skipped those rows and a few before and after, I have skipped the first 2000 rows, I have reordered the rows before they are processed, but the code always breaks on the same number it process.

       

      My code it output is this, The ArrayList of Field is simply a list of my columns with Field containing only the column name and column type:

      private void insertDataCSV(Table table,
        String fileName,
        ArrayList<Field> queryItems) throws TableauException {
        TableDefinition tableDef = table.getTableDefinition();
               
        BufferedReader br = null;
        String line = "";
        String cvsSplitBy = ",(?=([^\"]*\"[^\"]*\")*[^\"]*$)";
              
        try {
        br = new BufferedReader(new FileReader(fileName));
        int x = 1;
        Row row = new Row(tableDef);
        while ((line = br.readLine()) != null) {
         
               // use comma as separator
        String[] elements = line.split(cvsSplitBy);
         
        int i = 0;
                for(Field item : queryItems){
                if(item.type == Type.CHAR_STRING){
                String utf8Value = "";
        try {
        utf8Value = new String(elements[i].substring(1, elements[i].length()-1).getBytes(Charset.defaultCharset()), "UTF-8");
        } catch (UnsupportedEncodingException e) {
        // TODO Auto-generated catch block
        System.out.println(e.getMessage());
        }
                row.setCharString(i, utf8Value);
                }
                i++;
                }
                try{
                table.insert(row);
                }
                catch(TableauException e){
                System.out.println(e.getMessage());
                }
                System.out.println(x + ": " + elements[0]);
                x++;
        }
         
        } catch (FileNotFoundException e) {
        e.printStackTrace();
        } catch (IOException e) {
        e.printStackTrace();
        } finally {
        if (br != null) {
        try {
        br.close();
        } catch (IOException e) {
        e.printStackTrace();
        }
        }
        }
        }