Lost!

Some people, like myself, test many CMS and ECM packages on a normal basis. There are, however, many people that may need to test these types of packages in a cheaper environment than on an enterprise class server that costs tens of thousands of dollars - whether they’re part of a small business, startup or even a large business with someone “testing” at home. On top of the CMS and ECM packages, I wanted something to help me test Enterprise class software (read: learn) for future use and I think I accomplished more than what I wanted.

Requirements
My requirements are completely over the top for someone testing an average CMS package like Drupal, Plone or even Joomla. My server is specifically setup for Enterprise class application testing.

I basically wanted something that would allow me to test SharePoint, Alfresco, and a whole host of other packages at once. I also wanted something to allow me to use Virtual Machines instead of just one OS. This allows me to physically turn machines on and off without even needing to be at the box. This will allows me to use products, side by side, in their own environments testing how memory and CPU resources effect the packages - I don’t need them trying to fight with each other on the same box.

My underlining OS is Windows 2008 Enterprise Server, 64bit. The reason I went with this instead of Linux and such, was because I needed to really need to sit down with 2008 and learn about how it works. I’ve also finally learned that businesses really prefer to pay for everything - especially on the government side of the line. (There was also an OS issue that I ultimately caused - it helped “guide” me to 2008 Server as well.)

Anyway - I am running another package that I’m learning about (running something you’re learning on is normally a very insecure thing - be careful during your planning) - Virtual Server 2005 R2 Enterprise. That will allow me to use 1 server for many. I decided on 3 base images of the OS I’m installing so I can power them up once a month, run updates and shut them down. If I need an image for something - cut’n'paste it and I’m pretty much fresh. (Note: There are some things that I’ve found that I don’t like about the MSFT VM Package - that story soon.) For machines, I decided on the following:

1. Windows 2003 R2 Enterprise Server: Not the “newest”, but the standard in most environments now. Right now, this is the only thing installed due to time constraints on my end.
2. Windows 2008 Server - Console Install: I was going to initially install this as the primary OS, but I didn’t want to pull my hair out trying to make everything work right while learning. For those that are unaware, MSFT has taken pointers from Linux on making servers headless for less overhead. Start this machine and all you get is a blinking curser, no GUI to log into.
3. Cent OS - This was planned as a Linux testbed for several applications, but there are weird issues that sprouted up. Ultimately, I have not had time to play with any Linux/Unix distros. This will happen soon though.

Now, that lists 3 servers - there are more. The extra servers are mirrors of their counterparts, just setup as separate Database Servers, Exchange Server or Images that are turned on once in awhile and updated. Right now, only the Windows side of things is setup. I’ve currently got IIS7, Exchange 2003 and MSSQL 2005 installed on 3 different machines. Yes, there is a reason for the older versions of Exchange and MSSQL… Thats for the VM Story later.

Now, the real question was - can I “cheaply” run all of those Virtual Machines on top of the physically installed OS? “Cheap” is a relative term. Over the course of the summer, “cheap” went from less than $1,000 to $2,500 to more than that. Now, this initial cost is just the hardware, it’s not counting the MSDN Premium package I purchased. In total, my budget was $5,000.

In terms of technical specifications, I required the following:

• Quad Core Server AMD or Intel - I didn’t care which. I’m not in the mood to compare Oranges and Tangerines.
• 4 2500GB+ Hard Drives in a RAID 10 array. Originally, I was going to use one 500GB drive for this exercise, but after thinking about it - it’s a Bad Idea©. The read/write on that one drive, with all machines turned on will be pretty bad, thus causing a higher likelihood of failure. RAID 10 was chosen over RAID 5 for performance reasons.
• 2 Onboard NICs. Mostly standard these days, but I needed to list it anyway.
• Memory - Ultimately, I got what I can afford. My requirements where a minimum of 8GB, but I wanted to see 16GB or more.
• Case, power, etc - Best option was a cheaper Barebones Upright case - I tore down my rack months ago.
• Graphics, for once, onboard VGA please.

Originally, I was going to put together this server for around $600, but have several physical machines. It can be done, if needed, but my requirements have changed and I actually got a job during researching the parts (which is why I don’t have time to write…) so things have changed. If you’re interested in this route, research Quad Core capable Shuttles. Low power, and small enough to get multiple machines in a tight area. Only downside - RAID will be hard to implement due to the space requirements.

The Server
My home server, otherwise known as the “TV” just wasn’t up to the task. While it has more Hard Drive space than any normal home computer should, it was designed specifically for what it does, a home PVR. There really isn’t a heck of a lot of processing power and memory to allow for more services, especially your average ECM package but when it comes to drive space, 2.5TB is overkill for a ECM test server as well. (TV for military in Europe is usually a season behind and a whole lot of re-runs, this’ll go to 5+ TB when we go back to the US.)

The first thing I did was hop onto eBay to check out the 1u and 2u servers offered. While there are thousands of servers out there, the “affordable” ones are either older or missing Memory and/or the CPU nearly all had no hard drives. In Enterprise class hardware, the drives (usually iSCSI) are really expensive, adding to the cost. On top of this, a lot of people flat out refuse to deal with APO based customers (Read: Military overseas) even though we have US based addresses - something that still really pisses me off. Anyway, after some soul searching, I ditched my hopes of finding a cheap, up-to-date, 1u Server. I decided to start searching for hardware built by smaller and larger shops - Dell, HP, IBM and even several shops listed in LinuxJournal - all too expensive. I couldn’t spend $5,000 on a test server that’s supposed to sit next to my desk, that was my full budget.

After some searching, I realized that no computer shop was going to be able to compete with a home-built server cost. While I won’t get support for broken parts, the warranty will cover that issue for the first few months - which is typically when things die. As for the hard drives, they die all the time. So, after thinking, I hopped back on NewEgg. Use what you will, but they’re the fasted and cheapest thing that I can get at my current location.

It turns out, NewEgg offers some very nice Asus servers. Here’s my hardware rundown:

• The Barebones Server: ASUS TS500-E4/PA4 Pedestal.
• Memory: 12GB FB-DIMM (6×2gb), brand doesn’t matter. Would have gone for 16+gb, but the cost($800) far outweighed the gains.
• Hard Drives: 2×80GB drives in RAID1 Array for main OS. 4×250GB drives in RAID10 Array (built in controller). All drives are standard SATA 3GB/s.

Drive trays: Since the main OS drives are in a RAID 1 array, I figured it would be best to have them cooled and removable. These ICY Dock trays show me the drive temps, have adjustable fan speeds, drive alarms and are decently cheap. Oddly, both docks had no problems compared to the reviews listed for it.

RAID 1 Array: This is the one thing I skimped on that I wish I wouldn’t have…that’s the reason for the “Almost” in the title. I’m running a Promise PCI RAID 1 card. The card runs perfectly fine, no problems. The problem comes from the fact that this is considered a Fake RAID card. It’s a hardware based, software RAID solution, very similar to what is on motherboards. If I had to do it all over, I would buy a full RAID1 card with battery backup, but it’s far too late for that unless this stinker comes crashing down.

Kicking The Tires
I must say, I’m very pleasantly surprised on how well this machine performs, even with the fake raid card in it.

When I first fired it up, it gave me a scare due to how long it takes for the POST to happen. Normally, I don’t worry about such things with large amounts of memory (if you’ve never dealt with a server, it is not to be thought of as a Ferrari, but more like a Mack truck…), but the screen remained blank for a bit longer than I anticipated.

Other than the startup time, which is a good 2.5 minutes, it runs Windows Server 2008 like a champ.

The Good
This thing runs strong. So far, I have no complaints about the hardware, cost or anything related to the physical machine other than minor details.

The OS side of things was a little iffy, so I’ll list that in the Bad section. But, I will say that Windows 2008 runs like a champ on this thing - no problems whatsoever.

On the noise front, this machine is quiet. It’s just a tad louder than the fan for my laptop cooling plate.

Here’s my Full NewEgg Review that’s listed on their product page:

Pros: Nicely built, very solid, no buzzing, etc. It’s great that it comes with a memory cooling unit as well. Cables already routed for you, clean inside.
Cons: -The 5 1/4 bays are just a tad cramped (.5mm or so), so certain things can be a bear to get in or get out if needed. For me, the middle bay, under the DVD drive was just a tad too small for a removable drive bay - had to play with it to get it in.

-No front fan and no easy way to mount one.

-Case intrusion switch can be a pain to reset. Remove battery, bridge reset pins and hold reset button for 15 seconds otherwise it’ll be a huge frustration for you. You can also just bridge the intrusion pins on the MB to not use the “feature”

-Not really a con, but the processors sit very close to the drive cage. Was pretty tight trying to get the fans on.
Other Thoughts: -Some may consider it loud, but if they’ve been in a Data Center, this noise is standard. It’s actually less noisy than it’s Dell counterparts after spin-up.

-Uses FB-DIMMs instead of normal ECC type.

-Does POST/Mem Check with no screen during fan spin-up, so don’t freak out when you first turn it on, give it a minute.

-Had one SATA cable that was bad.

Running Dual Quad XEONs E5410 2.33GHz, 12GB memory, RAID 10 on the drive cage plus RAID 1 off a Promise PCI card. Runs nicely as a VM server for web dev.

The Bad
While it’s more of my fault, due to budget constraints, I really wish that I could have gone with more than 12Gb of memory. Right now, If I turn on each machine with the max memory allowed for each, I can get a total of 4 machines running at once, but it freaks everything out since I’m using 100% of memory. If you do this, you’ll have to play with memory allocation till you get things running smoothly. The main problem here is that FBDIMMs are REALLY expensive.

Operating systems on this machine just didn’t want to play well. I attempted to load several flavors of Linux on this machine and it was just Not Fun. FreeBSD didn’t like the FakeRaid card (my fault) and complained about it from the beginning, OpenSUSE crashed hard within a day of installation, Ubuntu installed, rebooted and lost itself, etc. I gave up on the Linux/Unix stuff due to the issues related to the FakeRaid card. On the Windows side of things, I installed Windows 2008 Enterprise and it works like a champ - no problems, no complaints, no nothin’. Again, this whole paragraph was my fault.

Benchmarks
I know a lot of people just live for benchmarks. Originally, I was planning on listing them, but I don’t have time to run and rerun benchmarks on it with VMs running, not running, stress testing, etc. I found a few tools for testing IIS, but don’t have time to play around with them. I’ll probably update this post in the next week or so (hopefully) with at least an IIS Stress Test.

Note: MSDN licenses are a cheap way to setup a development lab since you can get tens of thousands of dollars in software for a hugely steep discount, but…you cannot use this software for a production environment. Also, MSDN Educational licenses are even less money, but you cannot use programs associated with them to actually sell anything you create. Some people have found out the hard way that Visual Studio actually marks your files to which license was used to create the program.

Preview of the Next Blog Post: Switching to a Mac:
I’ve been a solid Windows user since ver. 3 (yes, that clunky windows version that everyone laughs at). Before then, I was a solid, hard core, DOS user. Over the years, I’ve used FreeBSD, Digital Unix, several Flavors of Linux, and all the Apple OS’ from 7.1 to X… This month, I made a major decision to completely switch to Mac, even though I do .Net dabbling, SharePoint work and all that Windows based stuff.