14 Replies Latest reply on Jun 25, 2018 2:13 PM by Toby Erkson

    Why does the "Java migrations" operation fail during a restore?

    Toby Erkson

      I've got 10.5.0 installed on my QA Server.  I have tried this on a 2-node set up (16 core primary, 8 core worker) as well as a single node setup (16 cores).  I get the same results.  During a restore of a 10.3.1 backup I get as far as "Running Java migrations" then it fails:

      At this point it then reverts to the prior state

      I am using the --no-config parameter.  The backup is from my single-node production server.

       

      What the heck are "java migrations"?

      Why is it failing?

       

      Attached file is the full error message from the command window.  Not sure if it will help.

        • 1. Re: Why does the "Java migrations" operation fail during a restore?
          Jeff Strauss

          Not sure, but I'll give it a try.  The message seen just prior to failing is: 

            *** Custom logo file D:/Application/Tableau/Tableau_Server/data/tabsvc/wgserver/images/custom/daimler_logo_big.png does

          not exist, skipping.

            *** Custom logo file D:/Application/Tableau/Tableau_Server/data/tabsvc/wgserver/images/custom/Daimler_Office_RGB_026mm.p

          ng does not exist, skipping.

            *** Custom logo file D:/Application/Tableau/Tableau_Server/data/tabsvc/wgserver/images/custom/small_logo.jpg does not ex

          ist, skipping.

          *** Error during restore: Component::InitError 'Unable to init component Components::Vizportal (7 of 16):

           

           

          Maybe Tableau is considering this a fatal kind of error that it can't find the custom logo files, the only reason I say this is because it's the vizportal process that is resonsible for displaying these logos.

           

           

          Can you try to set the logos to default values prior to the restore and then see if this at least helps?

          • 2. Re: Why does the "Java migrations" operation fail during a restore?
            Toby Erkson

            Looking through the ...\logs\tabsvc-tabadmin-java.log:

            ...

            2018-01-23 10:58:04.936 -0800 19388@QTNAICVDW064 main INFO  com.tableausoftware.dbmigration.MigrationDriver - Applying Migration20170906000001DeleteTriggerToBackfillSiteRoleIdFromUsersTableWhenNotSpecified.sql (20170906000001)

            2018-01-23 10:58:04.936 -0800 19388@QTNAICVDW064 main INFO  com.tableausoftware.dbmigration.MigrationDriver - Running in transaction: true

            2018-01-23 10:58:04.936 -0800 19388@QTNAICVDW064 main INFO  com.tableausoftware.dbmigration.MigrationDriver - Migration20170906000001DeleteTriggerToBackfillSiteRoleIdFromUsersTableWhenNotSpecified.sql (20170906000001) succeeded.

            2018-01-23 10:58:04.936 -0800 19388@QTNAICVDW064 main INFO  com.tableausoftware.dbmigration.MigrationDriver - Applying Migration20170906000002DeleteOldSiteRoleColumnsFromUsersTable.sql (20170906000002)

            2018-01-23 10:58:04.936 -0800 19388@QTNAICVDW064 main INFO  com.tableausoftware.dbmigration.MigrationDriver - Running in transaction: true

            2018-01-23 10:58:04.952 -0800 19388@QTNAICVDW064 main ERROR com.tableausoftware.dbmigration.MigrationDriver - Migration Migration20170906000002DeleteOldSiteRoleColumnsFromUsersTable.sql (20170906000002) failed.

            org.postgresql.util.PSQLException: ERROR: cannot drop table users column admin_level because other objects depend on it

              Detail: view _security_administrators depends on table users column admin_level

              Hint: Use DROP ... CASCADE to drop the dependent objects too.

                at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)

                at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)

                at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)

                at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)

            ...

             

            Which appears to correlate with ...\logs\notify-tabadmin.log:

            ...

            2018-01-23 10:58:04.967 -0800_FATAL_170.2.62.20:QTNAICVDW064_:_pid=19388_0x5ad5be4a__user=__request=__ Failed to run Java migrations: com.tableausoftware.dbmigration.MigrationException: Migration failed: Migration20170906000002DeleteOldSiteRoleColumnsFromUsersTable.sql (20170906000002).

            Backtrace: com/tableausoftware/dbmigration/MigrationDriver.java:159:in `migrate'

            com/tableausoftware/dbmigration/MigrationDriver.java:130:in `migrate'

            com/tableausoftware/dbmigration/MigrationDriver.java:119:in `migrateTo'

            com/tableausoftware/dbmigration/MigrationDriver.java:105:in `migrateAll'

            file:D:/Application/Tableau/Tableau_Server/10.5/bin/tabadmin.jar!/lib/service/java_db_migration.rb:38:in `run_migrations'

            ...

            • 3. Re: Why does the "Java migrations" operation fail during a restore?
              Toby Erkson

              This 2-node testing QA Server already has these images on it.  The file path is the same for 10.3.1 & 10.5.0.  The fact that the process skips the files leads me to believe this isn't the issue.

               

              This 2-node QA Server was originally 10.3.1 and had the PROD Server 10.3.1 backup restored on it.  Works.  I wonder if I should down-grade this 2-node QA Server to 10.3.1 and then try to restore the current 1-node QA Server 10.3.1 backup onto it.  Ugh

               

              Here's a visual to help the explanation

              • 4. Re: Why does the "Java migrations" operation fail during a restore?
                Toby Erkson

                I can't downgrade at this time as this 2-node QA Server has another issue:  It can't display workbook views   Already have a ticket w/Support on it.

                 

                Putting this issue on hold until the above issue gets resolved.

                 

                Image result for on hold

                • 5. Re: Why does the "Java migrations" operation fail during a restore?
                  Dan Scott

                  Toby,

                  I can give a few answers. First off, as you may have realized from the error messages you showed, the "Java migrations" referred to are really just database migrations that act on the Repository (PostgreSQL database) to update its structure and contents. This used to be handled in Tableau via "Ruby migrations" that used "ActiveRecord." The use of ActiveRecord was an artifact of Tableau Server's early years as a Ruby on Rails application. We have moved away from that and now we use a Java-based database migration system. Hence the (possibly confusing) "Java migrations" name. Again, these are really *database* migrations.

                   

                  In this case, the error message you posted explains what is wrong and suggests what you may be able to do to correct the problem. Quoting from your tabadmin-log.java error message:

                   

                  ERROR com.tableausoftware.dbmigration.MigrationDriver - Migration Migration20170906000002DeleteOldSiteRoleColumnsFromUsersTable.sql (20170906000002) failed.

                  org.postgresql.util.PSQLException: ERROR: cannot drop table users column admin_level because other objects depend on it

                    Detail: view _security_administrators depends on table users column admin_level

                    Hint: Use DROP ... CASCADE to drop the dependent objects too.

                   

                  One of the lines in the migration file is trying to drop the admin_level column from the users table. The migration in question looks like this:

                   

                  -- Drop old columns from users table represented site role admin_level, licensing_role_id, publisher_tristate

                  ALTER TABLE users

                    DROP COLUMN IF EXISTS admin_level,

                    DROP COLUMN IF EXISTS licensing_role_id,

                    DROP COLUMN IF EXISTS publisher_tristate

                  ;

                   

                  Now, according to your error message, PostgreSQL is complaining that it cannot drop the admin_level column, as instructed, because there is a "view" defined in your database called _security_administrators, and the definition of that view *refers* to the admin_level row. So, if it dropped it, the view wouldn't make any sense. Hence Tableau's migrations came to a screeching halt.

                   

                  The error message (which originates with PostgreSQL, itself) is suggesting one possible remedy. It says, "Hey, you *could* blow past this if you just tell me to *also* delete the _security_administrators view (which can be done using "CASCADE") to cascade the deletion of the column on to include deleting the view. Even if correct, though, that would be problematic for you, since the contents of the migration were set by Tableau, and are not readily messed with.

                   

                  Anyway, there is an important question to ask, before deciding what to do: "Did someone at your organization create this view? Perhaps as a convenience?" Because, so far, I can't see any sign of the view being defined by Tableau---though it does follow a Tableau convention of naming views with a leading underscore. Don't take that as a definitive statement though. I'm away form work and it's very late... But, *if* this was defined on your end, then the solution will probably be pretty simple: You would just run the database and issue it a command to drop the view: DROP VIEW _security_administrators;  Then, the next time the migrations are run, it would have nothing to complain about, and it should succeed (barring other unanticipated landmines).  Note: if you *did* define this view (rather than Tableau) it is possible that you might have some processes that depend on it working the way it does, so you would want to understand what might then become broken when it goes away.

                   

                  If this view was defined by Tableau, then that would be a bit of a head-scratcher. But, in such a case, I would generally expect lots and lots of customers to have the same problem, and I am not aware of anything like that.

                   

                  So, if you can get a version of Tableau Server launched (so the database is running) then you can drop the _server_administrators view, as I explained, and then try the upgrade after that.

                   

                  But, given that you seem to have some other issues going on, working with Tableau Support to get through things (and make sure there are no mistakes) would probably be wise. You can send them my post, as they will likely find it helpful.

                   

                  -Dan-

                  2 of 2 people found this helpful
                  • 6. Re: Why does the "Java migrations" operation fail during a restore?
                    Toby Erkson

                    Dan,

                    First, thank you for the detailed reply, that is AWESOME info!

                    Second, this triggered my memory:

                    Anyway, there is an important question to ask, before deciding what to do: "Did someone at your organization create this view? Perhaps as a convenience?" Because, so far, I can't see any sign of the view being defined by Tableau---though it does follow a Tableau convention of naming views with a leading underscore.

                    Yes, that is a Tableau naming convention and was added & used by me several versions ago to build a workbook that contained a comprehensive workbook viz for Tableau Server permissions.  There were actually three views created that I will need to now delete:

                    _security_all_permissions

                    _security_permissions

                    _security_administrators

                     

                    I didn't realize that these views persisted!   And now it's bitten me on the ****!  Well, it's okay to drop these as they are no longer used and haven't been for some time.  I can do this.  I'm pretty sure, at least...

                     

                    It is interesting that this has never been an issue until now.

                    • 7. Re: Why does the "Java migrations" operation fail during a restore?
                      Dan Scott

                      Toby,

                      That's excellent news! One approach to this might be to delete the unnecessary views from your production database, then take another backup, and restore that backup to your QA environment using a clean reinstall of 10.5, so it doesn't attempt to use leftover data from the failed upgrade attempt. I'm pretty sure that approach would work, but there may be easier options that would take less time---However, someone else would probably give better advice about that, since this isn't something I have a lot of expertise on.

                       

                      Note: it's no great surprise that this hasn't created problems until now. As long as the database schema remained compatible with what your views expected, then everything would remain okay. This just happens to be the first time that an incompatible change was made.

                       

                      I'm really glad to hear that the mystery has been cleared up, and I think you are poised to start moving forward again! :-)

                       

                      -Dan-

                      • 8. Re: Why does the "Java migrations" operation fail during a restore?
                        Toby Erkson

                        Success!

                        I was able to log into the postgres db and drop the three offending views on the 1-node QA Server.  I then ran the backup command (I kept the current one just in case, of course).  Next, I executed my batch script to restore the 2-node QA Server with the new 1-node backup file.  Here is the output but edited to exclude the constant & repetitive warning messages(1):

                        18:39:01  D:\Application\Tableau\Scripts (yes, I have a custom cmd PROMPT )

                        >tabadmin stop

                        ===== Stopping service...

                          -- Service stopped successfully

                         

                        18:42:22  D:\Application\Tableau\Scripts

                        >tabadmin restore --no-config "\\corpnas02\tableau\daily_backup_QA_2018-01-24.tsbak"

                        ===== Beginning to restore the backup

                        ===== Verifying backup manifest

                        Please enter the password for mydomain\RunAsUser:

                        ===== Initializing Tableau Server File Store remotely

                          -- Initiating built-in extract engine data restore

                          -- Restoring service data from backup file

                          -- Database restore completed.

                          -- running migrations

                        ===== Running Java migrations

                        ===== Validating Database schema signature

                        ===== Database schema is different from the expected schema. Use tabadmin validate_schema_signature for further investigation.

                        ===== User language: en

                        ===== Initializing Tableau Server Coordination Service 0 remotely

                          -- Waiting for built-in extract engine data restore to finish.

                          -- Extract engine data restore completed.

                        ===== Setting inheritance on D:/Application/Tableau/Tableau_Server/data/tabsvc/pgsql/data

                        ===== Setting inheritance completed on D:/Application/Tableau/Tableau_Server/data/tabsvc/pgsql/data

                        ===== Sos restore completed

                        Number of restored objects: 0

                        ===== Setting inheritance on D:/Application/Tableau/Tableau_Server/data/tabsvc/pgsql/data

                        ===== Setting inheritance completed on D:/Application/Tableau/Tableau_Server/data/tabsvc/pgsql/data

                          -- Restoring web data connectors

                          -- Web data connectors were distributed to all gateways.

                          -- Web data connectors restore completed.

                        ===== Building search index

                        ===== Search index built

                        ===== Setting inheritance on D:/Application/Tableau/Tableau_Server/data/tabsvc/pgsql/data

                        ===== Setting inheritance completed on D:/Application/Tableau/Tableau_Server/data/tabsvc/pgsql/data

                        ===== Backup restore completed

                         

                        Restarted the 2-node QA Server and it works, looking just like the 1-node QA Server.  Yeay!  Another step closer

                         

                        Thank you Dan, I couldn't have done it without your invaluable help

                         

                         

                         

                         

                        (1)

                          *** WARNING: Invalid or inadvisable value(s) found in: 'D:/Application/Tableau/Tableau_Server/config/tabsvc.yml'

                        ** Configuration will be completed with the requested values for the following options, but please note the warnings:

                        The value of '8' for 'worker1.backgrounder.procs' is not recommended. 8 instances exceeds 4, which is the maximum recommended number of backgrounder processes. Consider lowering the number of processes to 4 or less.

                        • 9. Re: Why does the "Java migrations" operation fail during a restore?
                          Dan Scott

                          Awesome! Happy exploration of 10.5. :-)

                           

                          -Dan-

                          • 10. Re: Why does the "Java migrations" operation fail during a restore?
                            Dan Scott

                            Oh, and don't forget to also drop your views from the production server before you try to upgrade it. In fact, I would suggest taking care of that more or less immediately, while you still have it at the top of your mind, and have just practiced it in QA.

                             

                            -Dan-

                            • 11. Re: Why does the "Java migrations" operation fail during a restore?
                              Toby Erkson

                              Yup, it's on my list of things to do this morning

                              • 12. Re: Why does the "Java migrations" operation fail during a restore?
                                Toby Erkson

                                I can't downgrade at this time as this 2-node QA Server has another issue:  It can't display workbook views   Already have a ticket w/Support on it.

                                This issue has now been resolved.  Don't know why but we'll attribute it to me muckin' around in the postgres db.  Some of us just gotta touch the hot stove

                                • 13. Re: Why does the "Java migrations" operation fail during a restore?
                                  Eric O'Bryant

                                  How did you drop the  view _security_administrators?  I believe I'm having the same issue.  Every time I try to upgrade to new server version say 10.5 or 2018.1.1 all looks good until the very end of the process and I fet a failed to initialize error.  So I have to restore from my backup.

                                   

                                  Thanks,

                                  Eric

                                  • 14. Re: Why does the "Java migrations" operation fail during a restore?
                                    Toby Erkson

                                    Eric, that view does not exist in a standard Tableau Server installation.  That was a custom view created back in version 9.  I recommend you work with Tableau Support regarding your particular issue.

                                    1 of 1 people found this helpful