5 Replies Latest reply on Feb 3, 2020 3:20 PM by Adrian Vogelsgesang

    How to change logging level in hyperd.log file

    Lindsay Weir

      Has anyone been able to change the logging level from the default INFO level in the hyperd.log file?

       

      The 'hyperd --settings' indicates that there are different log levels that can be applied but I have not been able to work out how to change these using the Python API.

       

      NAME                                         | DEFAULT                                                                                                   | SCOPE   | DESCRIPTION
      Hyper.Mode.Experimental                      |                                                                                                           | Global  | Sets Hyper to Experimental Mode
      Hyper.Mode.Standard                          |                                                                                                           | Global  | Resets all settings affected by `Hyper.Mode.Experimental`
      Tableau.LogLevel.Debug                       |                                                                                                           | Global  | Activates settings for Tableau logging level `debug`
      Tableau.LogLevel.Error                       |                                                                                                           | Global  | Activates settings for Tableau logging level `error`
      Tableau.LogLevel.Fatal                       |                                                                                                           | Global  | Activates settings for Tableau logging level `fatal`
      Tableau.LogLevel.Info                        |                                                                                                           | Global  | Activates settings for Tableau logging level `info` (default)
      Tableau.LogLevel.Off                         |                                                                                                           | Global  | Activates settings for Tableau logging level `off`
      Tableau.LogLevel.Trace                       |                                                                                                           | Global  | Activates settings for Tableau logging level `trace`
      Tableau.LogLevel.Warn                        |

       

       

      I have tried using:

      hyper = HyperProcess(Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU, parameters={"config":"configfile"})

       

      and setting various options in the file 'configfile' but nothing seems to set it.

      Any ideas as the log file grows to be multiple Gb in size each day with the volume of data I am trying to load.

        • 1. Re: How to change logging level in hyperd.log file
          Jonas Kammerer

          Edit:

          Hello!

           

          Since the "Hyper API 0.0.10309" release, the size of log files can be controlled with settings that are described in the process settings section of the HAPI sql documentation.
          {{"log_file_size_limit", "100m"}, {"log_file_max_count", "2"}}

           

          This should make it such that you only have two log files that will not grow larger than 100 megabyte. After two 100mb files exist, the older one will be deleted automatically and a new one will be created.

           

           

          Original (outdated) Post:

          Hello!

           

          We are planning to document logging settings for the next HAPI release. Until then you can try those two settings:
          {{":log_file_size_limit", "100m"}, {":log_file_max_count", "2"}}

           

          This should make it such that you only have two log files that will not grow larger than 100 megabyte. After two 100mb files exist, the older one will be deleted automatically and a new one will be created.

           

          Note that this will break with the next HAPI release however, since we plan to slighly adjust setting names.

           

           

          I hope this helps.

          • 2. Re: How to change logging level in hyperd.log file
            Adrian Vogelsgesang

            Hi Lindsay,

             

             

            Be careful with the Tableau.LogLevel settings, as those will be removed in an upcoming release of HyperAPI. Even today, they are not really useful. Instead, the setting you are probably looking is "log_config"...

             

            But before we go into details on this, could I ask you for more details on your use case?
            We are trying to log just the right amount of data by default, so that fiddling with our log settings shouldn't be necessary. Hence, I would be interested why you would want to adjust log settings - maybe there is something we should improve in our logging defaults, so that the out-of-box default settings work better also in your use case

             

            Are we logging too much or not enough?
            If too much: how much log volume do you actually get? Do you happen to know which messages are too verbose for your use case?
            If not enough: which additional information would you be looking for?

             

            Cheers,
            Adrian

            • 3. Re: How to change logging level in hyperd.log file
              Adrian Vogelsgesang
              Any ideas as the log file grows to be multiple Gb in size each day with the volume of data I am trying to load.

               

              Internally, even when we are loading multiple Terabyte of data into Hyper, our log files usually don't reach more than a few MB.

               

              This makes me wonder: how are you actually loading your data?

               

              I have a feeling that your current approach of loading the data is inefficient, even independent of the amount of log data generated. Maybe, by changing your data ingesting approach, we can solve your logging issues and at the same time even improve your loading times

               

              The best practices to insert data are:

              • Don't use "INSERT INTO Extract VALUES(...)". It's really inefficient for larger data volumes
              • Don't recreate Inserter instances inside a loop. Creating an Inserter instance is costly.
              • Instead, you should create a single Inserter instance and use it for all your tuples

              A "perfect" HyperAPI application creates exactly one Inserter per table it wants to insert data into and reuses this Inserter for all tuples.

               

              Could you maybe share your HyperAPI script with us? (If you prefer not to post it publicly here in the forum, you can also send it to me via emai)

               

               

              Cheers,
              Adrian

              • 4. Re: How to change logging level in hyperd.log file
                Lindsay Weir

                Thanks Jonas and Adrian for your responses.

                 

                For some background, I am definitely getting too much logging based on my use case so at least i looks like I can limit it based on Jonas' suggestion there until the next API release and the changes there.

                 

                I am working on pulling streaming data from a Kafka topic which includes geometry data. Periodically I need to close the hyper file and publish that to the server.

                As part of that process (due to the limitations of not being able to insert wkt geometries natively into the table), we have to insert those into a temporary table and then do the insert into so we can map the WKT into the GEOGRAPHY type. Each time we have to manipulate the tables (create temp table, insert into table from temp table, drop temp table, close file) in order to use the client server API to publish to the server. I think all these steps are adding to the excessive logging.

                 

                We also need to make sure the Schema/Table name used are Extract.Extract for the publish function to support it without then having to convert the hyper file into a .tde.

                 

                Good to know about the Inserter function as I might have that in a nested function so I might try moving that out of there to see if I get better performance as well.

                • 5. Re: How to change logging level in hyperd.log file
                  Adrian Vogelsgesang

                  As part of that process (due to the limitations of not being able to insert wkt geometries natively into the table), we have to insert those into a temporary table and then do the insert into so we can map the WKT into the GEOGRAPHY type. Each time we have to manipulate the tables (create temp table, insert into table from temp table, drop temp table, close file) in order to use the client server API to publish to the server. I think all these steps are adding to the excessive logging.

                  Well, this is indeed annoying, and we are currently looking into ways to improve this workflow

                   

                  Good to know about the Inserter function as I might have that in a nested function so I might try moving that out of there to see if I get better performance as well

                  Please let us know how things turn out. It would be interesting probably not only to us but also to other forum members if/by how much avoiding recreation of Inserters improves insertion performance and log size.

                   

                  Cheers,
                  Adrian