Announcement

Collapse
No announcement yet.

An Exercise in Virtualization

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • An Exercise in Virtualization

    This is a bit of a long story, documented here for the sharing of it and hopefully to spur some discussion.

    So I've been a long time advocate of virtualization, especially in the test and development world that I have lived in. I've had a home server of sorts for the last eight years, but it was only a stand alone server. At one point I installed one of the layered virtualization products (I think it was the original VMWare GSX Server). A little over two years ago, I built out a 'whitebox' home server and put Microsoft Hyper-V on it. That worked fairly well, but I had some stability issues and I found the base Windows 2008 installation consumed too much resource.

    A few weeks back, I was in TeamSpeak with Warlab and Apophis and we were discussing virtualization. Apophis was describing some of the nifty stuff he gets to play with and Warlab was lamenting how much he'd like to use it in his work. But all of this got me thinking and I began to investigate what it would require to get the VMWare ESXi virtualization platform running on my little whitebox server.

    I did some digging around online and found a website that describes the hardware compatibility of ESXi with popular commodity hardware (http://www.vm-help.com/esx40i/esx40_whitebox_HCL.php). I figured out pretty quickly that my existing RAID solution (A Highpoint 4-port PCIe 4x card) would not be supported. CPU driven RAID solutions such as the ICH10R chipset work but you don't see the raid set, only the individual drives. Network card support is limited and the onboard Marvel NIC wouldn't work. So I started looking around on eBay for devices guaranteed to be compatible with ESXi 4.1.

    The first card I bought was an Adaptec 2610SA RAID controller. PCI 64-bit, 6-port SATA RAID. Sounded perfect, and it was $20 on eBay. Bad news was... it only supports SATA 1.5Gb/s, doesn't support drives larger than 1TB, and you can't make RAID arrays larger than 2TB. Well, that's going back on eBay.

    Next, I bought an Intel Pro/1000 MT PCI Desktop adapter. $15 on eBay, brand new, works perfectly. For ESXi 4.1, they dropped support for both the 3Com 905 series and the Intel Pro/100 series. The Intel Pro/1000 (or anything that uses the e1000 driver) is the best bet for a whitebox.

    Now I needed a replacement for the failed Adaptec 2610SA purchase. I started to investigate the HP E200 controller. It's PCIe 4x, which is fine because my motherboard has two PCIe 16x slots. It's hardware RAID, so ESXi 4.1 will view the arrays. You need a 128MB batter backed up cache addon card, but a lot of the sellers will include it. It supports 8 SATA/SAS drives, but you need special cables to make the connection (4 drives per cable). It's a full length PCIe card, but my case can handle that. The hardest part was finding the cable part numbers, but once I did I discovered they were about $18 on eBay. Done. Purchased. I thought I'd done the research, but I kept digging (paranoid now).

    Closer reading shows that while this card works with ESXi 4.1, it has less than desirable performance. Everyone who had one was recommended to upgrade to the HP P400. Same size card, but much better performance. It has a 512MB battery backed up cache, and uses the same SAS/SATA cables. $50 on eBay. Fine. Bought. E200 is going on Craigslist first, then onto eBay if it doesn't sell.

    So now I've got a whitebox server that looks like this, hardware wise:

    Antec Titan 650 server case
    Gigabyte EP45-UD3P motherboard
    Intel Q9400 2.66GHz processor
    8GB DDR2 Memory
    HP P400 PCIe hardware RAID controller
    Intel Pro/1000 NIC
    Seagate 1.5TB SATA drives (4x)

    So now to the installation. This, was actually the neat part. Registered with VMWare's website and downloaded the ESXi 4.1 ISO. Burned it to CD. I had an older 2GB USB 2.0 thumb drive that I wasn't using any more, so I plugged it into the back of the case. The BIOS treats it like a hard drive. Booting the CD, I was given an option to install ESXi to the thumb drive. Awesome, completely independent of the storage drives. Installation went easily and I naviated through the console interface to setup networking and passwords. Once all that was done, I connected to the web interface from my desktop (over the network) and downloaded/installed the vSphere client (to manage the ESXi installation).

    Here's where I ran into a problem. I had created a single 4.5TB logical drive on the RAID controller, and it wasn't showing up in the storage options in ESXi. I racked my brain for probably a good two hours on this until I found out why. ESXi 4.1 has a limitation that it won't work with logical drives larger than 2TB. So my 4.5TB logical drive got broken up into two 2TB drives and a 500MB-ish drive. Now they showed up in the storage options.

    Through the console to the server I had enabled Tech Support mode, which enabled SSH access (not normally turned in a real production environment). Using WinSCP, I copied the ISO files for Widnows 2008 Server R2 and Windows 7 64-bit. Two new virtual machines were created (one after another) and each of the ISO files were mounted and installed into the virtual machines easy peasy.

    So now I have a Windows 7 test platform that I can do my development work on. If I wanted, I could create a snapshot of the virtual machine at a point and be able to roll back any changes I made to that snapshot, which is great for testing. I also have a Windows 2008 R2 server, which I have Active Directory Services, Certificate services, Remote Application services and Terminal Services Gateway running for secure remote access and application publication. It's like Microsoft's version of Citrix, and it tunnels right through my employer's corporate firewall (which is how I'm posting this).

    Again, just through I'd share some of my struggles in this little experiment.

    "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

  • #2
    Re: An Exercise in Virtualization

    Great read! I'm glad things worked out for you.

    Even in production environments, you are better off having multiple smaller LUNs vs. fewer large LUNs when it comes to disk access. With VMware you'll want to keep your disks under 2TB. I believe it needs to be 1GB smaller than 2TB, so 2047GB per disk.

    Personally. I tend to keep them all around 1TB.

    It's also well worth giving ESXi a try even if your hardware isn't on the compatibility list. Lots of times hardware will work that isn't on the list, but there are always those occasions where you end up beating your head against a wall because of some small device driver that isn't compatible.

    Another thing to keep in mind is it can often times be difficult (especially with fiber channel interfaces) to add them to the host AFTER ESXi is installed. I try and get my hardware settled before I ever install the hypervisor. With that being said; it's also dead easy to reinstall ESXi completely if you do install new hardware. As long as you don't touch your disk pools, you can just import them right back into VM, browse the drives, and re-import all your machines.
    Diplomacy is the art of saying "good doggie" while looking for a bigger stick.

    Comment


    • #3
      Re: An Exercise in Virtualization

      Wow, that's pretty advanced setup. Personally I don't have much of a need for something like that, but I could see it's use later on in game development.

      I only have one VM on my home machine under VirtualBox and it's a Ubuntu VM for mostly coding and SSHing to other servers.
      |TG-18th| Acreo Aeneas
      TG World of Tanks Clan Executive Officer
      Former 9th & 13th

      Pronounciation: Eh-Cree-Oh Ah-Nay-Ess
      Still can't say it? Call me Acorn then. -.-





      SSDs I Own: Kingston HyperX 3K (240 GB), Samsung 840 Pro (256 GB), Samsung 840 EVO (250 GB), Samsung 840 x 2 (120 GB), Plextor M5S (120 GB), OCZ Vertex (30 GB)

      TG Primer and Rules

      Comment


      • #4
        Re: An Exercise in Virtualization

        Damonte, I have an old Optiplex with 4gb Ram, 4x500gb hdds lined up so far.
        Going to be ordering a Raid controller in a week or so when I get back from vacation.
        Albeit at a slow pace, the ESXi project is underway :)
        Sorry I was so tied up in a game the other day that I didnt tell you that bro.

        |

        Comment


        • #5
          Re: An Exercise in Virtualization

          I have an Adaptec 2610SA RAID card for sale. Ha!

          "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

          Comment


          • #6
            Re: An Exercise in Virtualization

            Okay, a most frustrated update.

            Word to the wise (Warlab), do NOT purchase any HP RAID controller cards unless you actually have an HP ProLiant server to put them in. None of HP's firmware update utilities will work unless it detects the card in an appropriate HP server. My HP E200 works fine, but it's dog slow to write. The P400's (I actually have two), they are detected on boot but they hang the system every time I go into the bios or go into the Array Configuration Manager. Hours of frustrated trying this and trying that, it all resulted in the cards going onto eBay (and thankfully selling quickly).

            From this experience I'm going with a RAID controller card branded from the manufacturer: LSI MegaRAID SAS8344ELP. It's an 8-port 128MB cache 3GB/s PCIe 4x card. eBay has them for $105 brand new (bulk package). I can get the breakout cables (the card has two plugs, each for four drives) for $20 on Newegg.

            Fortunately, it looks like the money I'll get from selling the E200, both P400's and the breakout cables will cover the cost of the LSI card.

            "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

            Comment


            • #7
              Re: An Exercise in Virtualization

              An update to this escapade, if nothing else but for posterity.

              The SAS8344ELP sold for almost $110 in fierce bidding on eBay. I wasn't quick enough to get it... but it turned out to be a blessing. Closer examination of the card revealed that while it has two connectors (each uses a 4-plug SAS cable), one is internal and one is external. That's definitely not what I need. A week later, a seller shows up on eBay with a dozen SAS8308ELP cards for about $100 a piece. These are the same as the 8344 but with two internal ports and no external ports. It also shows up on the ESX whitebox HCL list as fully compatible. I ordered it, recieved it and plugged it in to ensure it works (and it does). I ordered a 3ware cable from Newegg this past Monday... and it should be arriving today. Will update then.

              On a side note, I've determined that 8GB for a virtualization box is just not enough. Almost all VM's seem to need 2GB of memory at a minimum, so I'm already looking at an upgrade. Apparently I can get an X58 chipset motherboard that supports 24GB of memory (six slots x 4GB sticks). Tempting.

              "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

              Comment


              • #8
                Re: An Exercise in Virtualization

                Extra memory is always good, but you can overprovision memory.

                Most of the time if a server isn't doing anything too crazy, it's not using much memory at all. That can be borrowed by the others as needed. From a home perspective; avoid looking at physical:virtual as 1:1 in terms of memory use.

                Just so I can say it twice; extra memory is always good anyway...
                Diplomacy is the art of saying "good doggie" while looking for a bigger stick.

                Comment


                • #9
                  Re: An Exercise in Virtualization

                  You bring up good points about memory over-provisioning and for most home uses it would probably be enough.

                  For me, my work situation has changed recently and I need a large home virtualization lab to test and explore software. I need to build a complete Active Directory environment with SSCM, database, secure remote access and such. Lots of virtual machines. Lots of memory, but not a lot of CPU required.

                  "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

                  Comment


                  • #10
                    Re: An Exercise in Virtualization

                    Another update.

                    The cables arrived. I reattached my four 1.5TB drives, plugged in the LSI card, attached the cables and powered on. The controller card chirped somewhere in the realm of twenty times (took about twenty to thirty seconds) before the system POSTed, but once it did I was able to boot into the MegaRAID WebBIOS. It's a pretty slick graphical interface with mouse support build into the controller card. I setup the four 1TB logical drives and one 92GB logical with all the left over space.

                    Booting (after twenty chirps) into my existing ESXi 4.1 installation (running from a USB pen drive) went without a problem. The new card was detected just fine, the drives were found and I was able to mount them as storage points. The last logical drive (92GB) I'm going to use to house my ISO files for installation and such.



                    Huzzah! Phase one is complete. I highly recommend this controller card and cable combination.

                    Edit: The controller is fast! With WriteBack and disk caching enabled, I'm uploading my ISO files at 12MB/s.
                    Last edited by Damonte; 07-27-2011, 08:07 PM.

                    "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

                    Comment


                    • #11
                      Re: An Exercise in Virtualization

                      Uber geek excellence Damonte !
                      Keep posting the updates, I love reading this :)

                      |

                      Comment


                      • #12
                        Re: An Exercise in Virtualization

                        Damonte just fulfilled my dream for one awesome NAS box. :D (Though he's doing it for virtualization.)

                        Nice Damonte! How's the performance like now with the new card and cables?
                        |TG-18th| Acreo Aeneas
                        TG World of Tanks Clan Executive Officer
                        Former 9th & 13th

                        Pronounciation: Eh-Cree-Oh Ah-Nay-Ess
                        Still can't say it? Call me Acorn then. -.-





                        SSDs I Own: Kingston HyperX 3K (240 GB), Samsung 840 Pro (256 GB), Samsung 840 EVO (250 GB), Samsung 840 x 2 (120 GB), Plextor M5S (120 GB), OCZ Vertex (30 GB)

                        TG Primer and Rules

                        Comment


                        • #13
                          Re: An Exercise in Virtualization

                          The performance is amazing. I enabled SSH (Remote tech support mode) so that I can SCP ISO files to the ESX datastores, and it's so quick. Creating and updating virtual machines is swift. This is an awesome controller card.

                          "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

                          Comment


                          • #14
                            Re: An Exercise in Virtualization

                            Have you tried several simultaneous (at least medium to large files) write and reads from several machines on your network to the NAS box?

                            (Something my future NAS would have to handle.)
                            |TG-18th| Acreo Aeneas
                            TG World of Tanks Clan Executive Officer
                            Former 9th & 13th

                            Pronounciation: Eh-Cree-Oh Ah-Nay-Ess
                            Still can't say it? Call me Acorn then. -.-





                            SSDs I Own: Kingston HyperX 3K (240 GB), Samsung 840 Pro (256 GB), Samsung 840 EVO (250 GB), Samsung 840 x 2 (120 GB), Plextor M5S (120 GB), OCZ Vertex (30 GB)

                            TG Primer and Rules

                            Comment


                            • #15
                              Re: An Exercise in Virtualization

                              I have not. It's not my intention to push this box that hard.

                              If you want to build yourself a really capable NAS, read this: http://forums.anandtech.com/showthread.php?t=2152079

                              There's lots of guidance out there for FreeNAS or OpenSolaris. They'll probably suit your needs better than my build.

                              "Everytime I read your posts I do it with Morgan Freeman's voice in my head as if he is narrating your life" - Aimed

                              Comment

                              Connect

                              Collapse

                              TeamSpeak 3 Server

                              Collapse

                              Advertisement

                              Collapse

                              Twitter Feed

                              Collapse

                              Working...
                              X