1 2 3 Previous Next 84 Replies Latest reply on Mar 30, 2015 6:23 AM by Cristian Vasile

    Would a RAM disk improve Server performance?

    Toby Erkson

      Just curious about this.  Your thoughts?

        • 2. Re: Would a RAM disk improve Server performance?
          Toby Erkson

          Nothing.  Just curious about this.  Cristian, I read your posts in the Forums regarding Desktop and was wondering how Server would be affected.

          • 3. Re: Would a RAM disk improve Server performance?
            Cristian Vasile

            OK. Short answer is yes.

             

            You have many tactical options.

             

            a) install a small ram disk on that machine, (8-10Gbytes) and alter Tableau's server TEMP variables to point to the ram disk. Test the PoC on a spare/testing machine!

             

            b) accelerate the entire windows partition where your .TDE files are located, using superspeed's supercache application http://www.superspeed.com/servers/supercache.php

             

            c) explore the smart SSD path. There are 3 zero costs applications able to evaluate your system and provide recommendations based on your IO load and disk access patterns.

            - ioturbine profiler http://get.fusionio.com/ioturbine-profiler

            - hgst profiler http://www.hgst.com/software/HGST-profiler

            - stortrends idata tool StorTrends® StorTrends iDATA Tool | All Flash Array & Hybrid Storage

             

            Could be more effective, less expensive to go with a "hybrid" approach, ie use a small ssd as an intermediate buffer and a "software accelerator" to dramatically improve  windows partition IO throughput like HGST ServerCache | HGST Storage (it works with any SSD) or Intel Cache Acceleration Software but your are bound to Intel manufactured SSD.

             

            d) add a PCI SSD enterprise board manufactured by Intel, Samsung, Fusion IO (owned by SanDisk), Virident/HGST (owned by WD), OCZ (owned by Toshiba) or Memblaze and move your Tableau files there.

             

            Hope this helps.

             

            Regards,

            Cristian

            • 4. Re: Would a RAM disk improve Server performance?
              Jeff Strauss

              I would love to hear if any of the above has a great impact on improved rendering or extract times.  Toby, do you plan to do a POC?

               

              My experience is that even with a truly optimal network / hardware / windows / Tableau configuration, speeds are only slightly faster.  Upon server rendering a view in isolation and then combing through the logs, what I observe is that requests are being cranked through quite efficiently, however being that 8.x.x processes requests sequentially and not much is shared, it takes longer than one would hope.  I have high hopes for nirvana with 9.x.x

              • 5. Re: Would a RAM disk improve Server performance?
                Cristian Vasile

                Jeffrey,

                 

                Below words are written by Allan Walker  on this thread Re: Hardware for the best Tableau performance

                Regards,

                Cristian.

                 

                ======================================================================

                Christian,

                 

                Just done some benchmarking with a ram disk.  I chose SoftPerfect Ram disk.

                 

                Number of marks: 5,500,000

                Mark type: Polygon

                 

                Package workbook on local hard drive: Time to go from opening file to fully render: 1 minute, 42 seconds

                Package workbook on (2GB NTFS) RAM drive: Time to go from opening file to fully render: 45 seconds

                 

                This is incredible.

                 

                Test data: Australian postcodes, csv files

                ======================================================================

                • 6. Re: Would a RAM disk improve Server performance?
                  Matt Coles

                  Cristian, have you tried (b) before? If so, what kind of an improvement did you see?

                   

                  Also, I'm curious about the (a) approach, as we have plenty of spare RAM to throw around (we just got new hosts for our Tableau Server cluster!). Right now my max temp folder on all hosts in the production cluster is 9GB. Say I create a 10 GB RAM disk and point Tableau Server at it via environment variable. What is the impact to the system should we run out of space on the RAM drive? Is Tableau Server smart enough to remove some older and unneeded files to make space without breaking anything? Is there a place we can specify the max size we wish to be used for temp storage?

                   

                  Thanks for all the detail. I'm interested in anything we can do to tune Tableau Server for better performance.

                  • 7. Re: Would a RAM disk improve Server performance?
                    Cristian Vasile

                    Matthew,

                     

                    Thank you for paying attention to this subject.

                     

                    Toby's question is a broader one, and he said that there is no issue. By the way he is a very experimented sysadmin, I know that he started his career writing code on assembler for Motorola 8 bit cpus. Also I clearly stated that he must run a PoC on a spare machine.

                     

                    Going back to your questions:

                    b) no, i did not run a test with supercache, but you could do this, if you have time and gear available, and post back the results.

                     

                    a) indeed, there is a risk to run out of space on not properly dimensioned RAM disk, but what happens if a production machine runs suddenly out of hdd space? I asked you because you are working for Tableau Software, INC

                     

                    Is Tableau Server smart enough to remove some older and unneeded files to make space without breaking anything? Is there a place we can specify the max size we wish to be used for temp storage?

                    I expect to have an answer to above questions for the finest coders hired by Tableau Software, INC who wrote the tableau stack, you could rise these issues on an internal trouble tickets system / email and then communicate back on forum.

                     

                    If your temp directory use ~9Gbytes being you I will create at least a 24Gbytes RAM disk, and in my Daily/Weekly/Monthly/Quarterly Functional Check List I should add a new task: check  empty space on RAM disk(s)

                     

                    Regards,

                    Cristian.

                    • 8. Re: Would a RAM disk improve Server performance?
                      Toby Erkson

                      Cristian,

                      First...how did you know I did 8-bit assembly coding?

                      Second, you're mostly right, I did teach myself assembler code but that was as a hobby in high school when the PC was first coming out.  So now we have an idea of how old I am

                      Third, buddy, you know more about this stuff than I do!  It's your conversation with Allan that got me thinking of this.  I'm glad you are contributing your knowledge with us   BTW, I asked Allan about the RAM disk and now that he's in 64-bit Tableau he no longer uses it.

                       

                      - Our environment is virtual and will be moving this year to a new and faster data center.  SSD isn't an option though it sounds really neat.  A hybrid approach does make sense.

                      - The option "b" you mention, "accelerate the entire windows partition where your .TDE files are located", I don't know if that would work for our situation.  The data that is used is diverse, meaning there isn't a handful of tables that everyone uses, we have hundreds of tables that are specifically used by one group that aren't used by any other group, and those other groups used tables not used by other groups, etc.  Now, if this block-level caching would help end users get vizes rendered more quickly then I can understand its usefulness.  You can see I don't know this technology so [please!] educate me if I'm wrong anywhere.

                       

                      Jeff,

                      I didn't plan on a PoC but I may try it on our QA machine as it seems easy enough.  I haven't done performance recordings (really haven't had a need to) so I'll have to get spun up on that as well as any other monitoring software y'all think I should use.  My background is coding and not server/hardware so my kung-fu is weak here than many of you   However, Matthew...

                       

                      Matthew,

                      I agree with Cristian that you could provide us with more info, more quickly, than by us non-Tableauistas experimenting and guessing at Tableau's boundaries.  If you have the RAM and the inside knowledge then what ever you can teach us would be awesome (<-- that's supposed to be a good icon ).

                      • 9. Re: Would a RAM disk improve Server performance?
                        Jeff Strauss

                        Assembler is really cool.  Seriously, the core knowledge provides a deep appreciation for internals, I wouldn't want to go back to it though as I still have flashbacks of having to debug code with all German variable names.  I did escape punch cards

                        • 10. Re: Would a RAM disk improve Server performance?
                          Cristian Vasile

                          Toby,

                           

                          I think that I saw an old post where you wrote something related to assembler language and 6800 or similar I do not remember now.

                           

                          Going back to your potential issue, things goes worse, you did mentioned magic word "virtual". The best bet in this case is to run some tests on QA environment with. Keep in mind that a VM could be migrated by the hypervisor from server A to server B , so any hardware device like a PCI SSD should be installed and deployed on all your bare metal servers. If you can afford a small SSD on each bare metal server and a smart cache layer, then could be a route to have happy internal customers, but is not the only one.

                           

                          I suppose that you collaborate with other technical personnel allocated to this migration tasks, especially with ones in charge with storage (SAN type, SAN real throughput, access protocols, dedicated networks for data "pipes") - it's very important to have a chat with them and let them know how often do you append/refresh data to .TDEs, yours business hours, SLA with business/departments (if you have such documents, etc).

                           

                          As you know, keeping data as close as possible to CPUs is the desired objective in your case.

                           

                          I contemplate again Jeffrey's words, and he is right, Tableau 9.x.x will deliver ccVizSQL, cc means Cache Coherent, an expression used in High Performance Computing, so maybe just switching to new tableau stack could suffice.

                           

                          Hope this helps.

                           

                          Regards,

                          Cristian.

                          • 11. Re: Would a RAM disk improve Server performance?
                            Cristian Vasile

                            Toby,

                             

                            You are not old, your are just Too Old to Rock 'n' Roll: Too Young to Die!

                            Regards,

                            Cristian.

                            • 12. Re: Would a RAM disk improve Server performance?
                              Toby Erkson

                              LOL, Jethro Tull, woo-hoo!

                               

                              Good conversation...good to know about version 9.  Looking forward to it.

                              • 13. Re: Would a RAM disk improve Server performance?
                                Cristian Vasile

                                Toby Erkson & Matt Coles,

                                 

                                I started an email train, on the matter of a spanning ram disk over multiple machines, with Mr. Olof Lagerkvist the author of ImDisk ramdisk solution. He found a way to merge multiple ram disks created on different servers, in a large, logical one. His solution is described below.

                                 

                                Hope this helps.

                                Regards,

                                Cristian.

                                 

                                =================================================================================

                                I would say that it would be possible to create a solution with a service running on each server, then Arsenal Image Mounter running on one of the servers connecting to all the others and create a one virtual disk for each connections to a storage provider server. Then, on this one server, you could use standard Windows Disk Management features or Windows 8 Storage Spaces feature to create one large volume from all virtual disks combined. Then you can format and share that merged volume on that server to the network so that client computers connect by simply attach to a network share on that particular server.

                                 

                                I just made a quick test and this seems to work.

                                 

                                1. On each storage providing server, create a virtual disk device and then devio.exe to share it over network.

                                imdisk -a -o awe -s 100M -u 0

                                devio 9000 \\?\GlobalRoot\Device\ImDisk0

                                 

                                2. On the "merging server", you need to run Arsenal Image Mounter and create a merged volume.

                                ArsenalImageMounterControl.exe /add /proxy=ip /filename=server1:9000

                                ArsenalImageMounterControl.exe /add /proxy=ip /filename=server2:9000

                                etc. Run this for each server connection.

                                Then in Disk Management, initialize the virtual disks as dynamic, create a merged volume using all of the virtual disks, format it and share it.

                                 

                                If your project is not licensed under an AGPL v3 compatible license, contact Arsenal Recon directly regarding alternative licensing:

                                http://ArsenalRecon.com/contact/

                                 

                                3. On each client computer, map a network drive to this share from each client.

                                 

                                One thing I could change to simplify this could be to modify devio.exe so that it could share a memory area directly without the need to create an ImDisk virtual disk device first. That should be fairly easy. But the above steps work with existing tools.

                                 

                                Links:

                                devio.exe: http://ltr-data.se/files/devio.exe

                                Arsenal Image Mounter: https://github.com/ArsenalRecon/Arsenal-Image-Mounter

                                 

                                You are very welcome with donations to my other open source projects

                                http://ltr-data.se/donate.html

                                 

                                Regards,

                                 

                                --

                                Olof Lagerkvist

                                LTR Data                  Twitter: @LTRData

                                Phone:  +46 70-345 89 52  Skype:  live:ol

                                E-mail: olof@ltr-data.se  Web:    http://www.ltr-data.se

                                =================================================================================



                                *********************************************************************************

                                -- Devio 3.04

                                 

                                It is now possible to use devio.exe to directly share physical memory from a server for use with for example ImDisk on another computer. With latest devio.exe you can share 300 MB physical memory:

                                devio 9000 \\?\awealloc 300M


                                o You still need to have ImDisk installed so that you have access to the awealloc driver though.

                                o It is possible to load image files with syntax like \\?\awealloc\??\C:\path\image.img and similar

                                 

                                And then on a client computer:

                                imdisk -a -t proxy -o ip -f serverip:9000 -m #: -p "/fs:ntfs /q /y /v:remoteram"

                                 

                                Latest 32-bit devio.exe: http://ltr-data.se/files/devio.exe

                                Latest 64-bit devio.exe: http://ltr-data.se/f...win64/devio.exe

                                *********************************************************************************


                                =================================================================================

                                Hello Olof,

                                 

                                Thanks for the messages and efforts!


                                1) Just to be sure, on the machine where I run devio.exe (the provider of RAM) I must install also ImDisk, because devio tool needs AWE driver? Am I right? By the way, are you still use AWE on 64 bit systems?

                                 

                                2) devio.exe can handle let's say 32Gbytes of RAM?
                                devio 9000 \\?\awealloc  32768M


                                Thank you.

                                 

                                Regards,

                                Cristian.

                                =================================================================================


                                **********************************************************************************

                                1.

                                Yes, that's right. You need to install ImDisk there to be able to use the awealloc driver. Awealloc is used to allocate physical memory. There is no difference between 32 bit and 64 bit versions in this case.

                                 

                                The 32 bit specific thing about using awealloc is that since it allocates physical memory, it can be used to allocate beyond the 4 GB RAM that for example standard versions of Windows Server can normally use. This means that using awealloc you can allocate several GB of memory even on 32 bit systems provided you have enough memory available and that you are running a server edition of Windows.

                                 

                                2.

                                Yes, that should be perfectly fine.

                                **********************************************************************************


                                • 14. Re: Would a RAM disk improve Server performance?
                                  Toby Erkson

                                  Dang Cristian, all I can say is "Wow"!  Thanks for the info

                                  1 2 3 Previous Next