Thursday, December 15, 2011

LAN-party house: Technical design and FAQ

After I posted about my LAN-party optimized house, lots of people have asked for more details about the computer configuration that allows me to maintain all the machines as if they were only one. I also posted the back story to how I ended up with this house, but people don't really care about me, they want to know how it works! Well, here you go!

Let's start with some pictures...

Sorry that there are no "overview" shots, but the room is pretty small and without a fish-eye lens it is hard to capture.

Hardware

In the pictures above, Protoman is a 2U rackmount server machine with the following specs:

  • CPU: Intel Xeon E3-1230
  • Motherboard: Intel S1200BTL
  • RAM: 4GB (2x2GB DDR3-1333 ECC)
  • OS hard drive: 60GB SSD
  • Master image storage: 2x1TB HDD (RAID-1)
  • Snapshot storage: 240GB SATA-3 SSD

I'll get into the meaning of all the storage in a bit.

The other machines on the rack are the gaming machines, each in a 3U case. The specs are:

  • CPU: Intel Core i5-2500
  • GPU: MSI N560GTX (nVidia GeForce 560)
  • Motherboard: MSI P67A-C43 (Intel P67 chipset)
  • RAM: 8GB (2x4GB DDR3-1333)
  • Local storage: 60GB SSD

Megaman and Roll are the desktop machines used day-to-day by myself and Christina Kelly. These machines predate the house and aren't very interesting. (If you aren't intimately familiar with the story of Megaman, you are probably wondering about the name "Roll". Rock and Roll were robots created by Dr. Light to help him with lab work and housekeeping. When danger struck, Dr. Light converted Rock into a fighting machine, and renamed him "Megaman", thus ruining the pun before the first Megaman game even started. Roll was never converted, but she nevertheless holds the serial number 002.)

The gaming machines are connected to the fold-out gaming stations via 35-foot-long HDMI and USB cables that run through cable tubes built into the house's foundation. Megaman and Roll are connected to our desks via long USB and dual-link DVI cables. I purchased all cables from Monoprice, and I highly recommend them.

Network boot

Originally, I had the gaming machines running Ubuntu Linux, using WINE to support Windows games. More recently, I have switched to Windows 7. The two configurations are fairly different, but let me start by describing the parts that are the same. In both cases, the server runs Ubuntu Linux Server, and all server-side software that I used is free, open source software available from the standard Ubuntu package repository.

As described in the original post, the gaming machines do not actually store their operating system or games locally. Indeed, their tiny 60GB hard drives couldn't even store all the games. Instead, the machines boot directly over the network. All modern network adapters support a standard for this called PXE. You simply have to enable it in the bios, and configure your DHCP server to send back the necessary information to get the boot process started.

I have set things up so that the client machines can boot in one of two modes. The server decides what mode to use, and I have to log into the server and edit the configs to switch -- this ensures that guests don't "accidentally" end up in the wrong mode.

  • Master mode: The machine reads from and writes to the master image directly.
  • Replica mode: The machine uses a copy-on-write overlay on top of the master image. So, the machine starts out booting from a disk image that seems to be exactly the same as the master, but when it writes to that image, a copy is made of the modified blocks, and only the copy is modified. Thus, the writes are visible only to that one machine. Each machine gets its own overlay. I can trivially wipe any of the overlays at any time to revert the machine back to the master image.

The disk image is exported using a block-level protocol rather than a filesystem-level protocol. That is, the client sends requests to the server to read and write the raw disk image directly, rather than requests for particular files. Block protocols are massively simpler and more efficient, since they allow the client to treat the remote disk exactly like a local disk, employing all the same caching and performance tricks. The main down side is that most filesystems are not designed to allow multiple machines to manipulate them simultaneously, but this is not a problem due to the copy-on-write overlays -- the master image is read-only. Another down side is that access permissions can only be enforced on the image as a whole, not individual files, but this also doesn't matter for my use case since there is no private data on the machines and all modifications affect only that machine. In fact, I give all guests admin rights to their machines, because I will just wipe all their changes later anyway.

Amazingly, with twelve machines booting and loading games simultaneously off the same master over a gigabit network, there is no significant performance difference compared to using a local disk. Before setting everything up, I had been excessively worried about this. I was even working on a custom UDP-based network protocol where the server would broadcast all responses, so that when all clients were reading the same data (the common case when everyone is in the same game), each block would only need to be transmitted once. However, this proved entirely unnecessary.

Original Setup: Linux

Originally, all of the machines ran Ubuntu Linux. I felt far more comfortable setting up network boot under Linux since it makes it easy to reach into the guts of the operating system to customize it however I need to. It was very unclear to me how one might convince Windows to boot over the network, and web searches on the topic tended to come up with proprietary solutions demanding money.

Since almost all games are Windows-based, I ran them under WINE. WINE is an implementation of the Windows API on Linux, which can run Windows software. Since it directly implements the Windows API rather than setting up a virtual machine under which Windows itself runs, programs execute at native speeds. The down side is that the Windows API is enormous and WINE does not implement it completely or perfectly, leading to bugs. Amazingly, a majority of games worked fine, although many had minor bugs (e.g. flickering mouse cursor, minor rendering artifacts, etc.). Some games, however, did not work, or had bad bugs that made them annoying to play. (Check out the Wine apps DB to see what works and what doesn't.)

I exported the master image using NBD, a Linux-specific protocol that is dead simple. The client and server together are only a couple thousand lines of code, and the protocol itself is just "read block, write block" and that's it.

Here's an outline of the boot process:

  1. BIOS boots to the ethernet adaptor's PXE "option ROM" -- a little bit of code that lives on the Ethernet adapter itself.
  2. The Ethernet adaptor makes DHCP request. The DHCP response includes instructions on how to boot.
  3. Based on the instructions, the Ethernet adaptor downloads and runs a pxelinux (a variant of syslinux) boot image from TFTP server identified by DHCP.
  4. pxelinux downloads and runs the real Linux kernel and initrd image, then starts them.
  5. initrd script loads necessary drivers, connects to NBD server, and mounts the root filesystem, setting up the COW overlay if desired.
  6. Ubuntu init scripts run from root filesystem, bringing up the OS.

Crazy, huh? It's like some sort of Russian doll. "initrd", for those that don't know, refers to a small, packed, read-only filesystem image which is loaded as part of the boot process and is responsible for mounting the real root filesystem. This allows dynamic kernel modules and userland programs to be involved in this process. I had to edit Ubuntu's initrd in order to support NBD (it only supports local disk and NFS by default) and set up the COW overlay, which was interesting. Luckily it's very easy to understand -- it's just an archive in CPIO format containing a bunch of command-line programs and bash scripts. I basically just had to get the NBD kernel module and nbd-client binary in there, and edit the scripts to invoke them. The down side is that I have to re-apply my changes whenever Ubuntu updated the standard initrd or kernel. In practice I often didn't bother, so my kernel version fell behind.

Copy-on-write block devices are supported natively in Linux via "device-mapper", which is the technology underlying LVM. My custom initrd included the device-mapper command-line utility and invoked it in order to set up the local 60GB hard drive as the COW overlay. I had to use device-mapper directly, rather than use LVM's "snapshot" support, because the master image was a read-only remote disk, and LVM wants to operate on volumes that it owns.

The script decides whether it is in master or replica mode based on boot parameters passed via the pxelinux config, which is obtained via TFTP form the server. To change configurations, I simply swap out this config.

New setup: Windows 7

Linux worked well enough to get us through seven or so LAN parties, but the WINE bugs were pretty annoying. Eventually I decided to give in and install Windows 7 on all the machines.

I am in the process of setting this up now. Last weekend I started a new master disk image and installed Windows 7 to it. It turns out that the Windows 7 installer supports installing directly to a remote block device via the iSCSI protocol, which is similar to NBD but apparently more featureful. Weirdly, though, Windows 7 apparently expects your network hardware to have boot-from-iSCSI built directly into its ROM, which most standard network cards don't. Luckily, there is an open source project called gPXE which fills this gap. You can actually flash gPXE over your network adaptor's ROM, or just bootstrap it over the network via regular PXE boot. Full instructions for setting up Windows 7 to netboot are here.

Overall, setting up Windows 7 to netboot was remarkably easy. Unlike Ubuntu, I didn't need to hack any boot scripts -- which is good, because I wouldn't have any clue how to hack Windows boot scripts. I did ran into one major snag in the process, though: The Windows 7 installer couldn't see the iSCSI drive because it did not have the proper network drivers for my hardware. This turned out to be relatively easy to fix once I figured out how:

  • Download the driver from the web and unzip it.
  • Find the directory containing the .inf file and copy it (the whole directory) to a USB stick.
  • Plug the USB stick into the target machine and start the Windows 7 installer.
  • In the installer, press shift+F10 to open the command prompt.
  • Type: drvload C:\path\to\driver.inf

With the network card operational, the iSCSI target appeared as expected. The installer even managed to install the network driver along with the rest of the system. Yay!

Once Windows was installed to the iSCSI target, gPXE could then boot directly into it, without any need for a local disk at all. Yes, this means you can PXE-boot Windows 7 itself, not just the installer.

Unfortunatley, Windows has no built-in copy-on-write overlay support (that I know of). Some proprietary solutions exist, at a steep price. For now, I am instead applying the COW overlay server-side, meaning that writes will actually go back to the server, but each game station will have a separate COW overlay allocated for it on the server. This should be mostly fine since guests don't usually install new games or otherwise write much to the disk. However, I'm also talking to the author of WinVBlock, an open source Windows virtual block device driver, about adding copy-on-write overlay support, so that the local hard drives in all these machines don't go to waste.

Now that the COW overlays are being done entirely server-side, I am able to take full advantage of LVM. For each machine, I am allocating a 20GB LVM snapshot of the master image. The snapshots all live on the 240GB SATA-3 SSD, since the server will need fast access to the tables it uses to manage the COW overlays. (For now, the snapshots are allocated per-machine, but I am toying with the idea of allocating them per-player, so that a player can switch machines more easily (e.g. to balance teams). However, with the Steam Cloud synchronizing most game settings, this may not be worth the effort.)

Normally, LVM snapshots are thought of as a backup mechanism. You allocate a snapshot of a volume, and then you go on modifying the main volume. You can use the snapshot to "go back in time" to the old state of the volume. But LVM also lets you modify the snapshot directly, with the changes only affecting the snapshot and not the main volume. In my case, this latter feature is the critical functionality, as I need all my machines to be able to modify their private snapshots. The fact that I can also modify the master without affecting any of the clones is just a convenience, in case I ever want to install a new game or change configuration mid-party.

I have not yet stress-tested this new setup in an actual LAN party, so I'm not sure yet how well it will perform. However, I did try booting all 12 machines at once, and starting Starcraft 2 on five machines at once. Load times seem fine so far.

Frequently Asked Questions

How do you handle Windows product activation?

I purchased 12 copies of Windows 7 Ultimate OEM System Builder edition, in 3-packs. However, it turns out that because the hardware is identical, Windows does not even realize that it is moving between machines. Windows is tolerant of a certain number of components changing, and apparently this tolerance is just enough that it doesn't care that the MAC address and component serial numbers are different.

Had Windows not been this tolerant, I would have used Microsoft's VAMT tool to manage keys. This tool lets you manage activation for a fleet of machines all at once over the network. Most importantly, it can operate in "proxy activation" mode, in which it talks to Microsoft's activation servers on the machines' behalf. When it does so, it captures the resulting activation certificates. You can save these certificates to a file and re-apply them later, whenever the machines are wiped.

Now that I know about VAMT, I intend to use it for all future Windows activations on any machine. Being able to back up the certificate and re-apply it later is much nicer than having to call Microsoft and explain myself whenever I re-install Windows.

I highly recommend that anyone emulating my setup actually purchase the proper Windows licenses even if your machines are identical. The more machines you have, the more it's actually worth Microsoft's time to track you down if they suspect piracy. You don't want to be caught without licenses.

You might be able to get away with Windows Home Premium, though. I was not able to determine via web searching whether Home Premium supports iSCSI. I decided not to risk it.

UPDATE: At the first actual LAN party on the new Windows 7 setup, some of the machines reported that they needed to be activated. However, Windows provides a 3-day grace period, and my LAN party was only 12 hours. So, I didn't bother activating. Presumably once I wipe these snapshots and re-clone from the master image for the next party, another 3-day grace period will start, and I'll never have to actually activate all 12 machines. But if they do ever demand immediate activation, I have VAMT and 12 keys ready to go.

Do guests have to download their own games from Steam?

No. Steam keeps a single game cache shared among all users of the machine. When someone logs into their account, all of the games that they own and which are installed on the machine are immediately available to play, regardless of who installed them. Games which are installed but not owned by the user will show up in the list with a convenient "buy now" button. Some games will even operate in demo mode.

This has always been one of my favorite things about Steam. The entire "steamapps" folder, where all game data lives, is just a big cache. If you copy a file from one system's "steamapps" to another, Steam will automatically find it, verify its integrity, and use it. If one file out of a game's data is missing, Steam will re-download just that file, not the whole game. It's fault-tolerant software engineering at its finest.

On a similar note, although Starcraft 2 is not available via Steam, an SC2 installation is not user-specific. When you star the game, you log in with your Battle.net account. Party guests thus log in with their own accounts, without needing to install the game for themselves.

Any game that asks for ownership information at install time (or first play) rather than run time simply cannot be played at our parties. Not legally, at least.

Is your electricity bill enormous?

I typically have one LAN party per month. I use about 500-600 kWh per month, for a bill of $70-$80. Doesn't seem so bad to me.

Why didn't you get better chairs!?!

The chairs are great! They are actually pretty well-padded and comfortable. Best of all, they stack, so they don't take much space when not in use.

You can afford all these computers but you have cheap Ikea furniture?

I can afford all these computers because I have cheap Ikea furniture. :)

I had no money left for new furniture after buying the computers, so I brought in the couches and tables from my old apartment.

How can you play modern games when most of them don't support LAN mode?

I have an internet connection. If a game has an online multiplayer mode, it can be used at a LAN party just fine.

While we're on the subject, I'd like to gush about my internet connection. My download bandwidth is a consistent 32Mbit. Doesn't matter what time of day. Doesn't matter how much bandwidth I've used this month. 32Mbit. Period.

My ISP is Sonic.net, an independent ISP in northern California. When I have trouble with Sonic -- which is unusual -- I call them up and immediately get a live person who treats me with respect. They don't use scripts, they use emulators -- the support person is running an emulator mimicking my particular router model so that they can go through the settings with me.

Best of all, I do not pay a cent to the local phone monopoly (AT&T) nor the local cable monopoly (Comcast). Sonic.net provides my phone lines, over which they provide DSL internet service.

Oh yeah. And when I posted about my house the other day, the very first person to +1 it on G+, before the post had hit any news sites, was Dane Jasper, CEO of Sonic.net. Yeah, the CEO of my ISP followed me on G+, before I was internet-famous. He also personally checked whether or not my house could get service, before it was built. If you e-mail him, he'll probably reply. How cool is that?

His take on bandwidth caps / traffic shaping? "Bandwidth management is not used in our network. We upgrade links before congestion occurs."

UPDATE: If you live outside the US, you might be thinking, "Only 32Mbit?". Yes, here in the United States, this is considered very fast. Sad, isn't it?

What's your network infrastructure? Cisco? Juniper?

Sorry, just plain old gigabit Ethernet. I have three 24-port D-Link gigabit switches and a DSL modem provided by my ISP. That's it.

Why didn't you get the i5-2500k? It is ridiculously overclockable.

I'm scared of overclocking. The thought of messing with voltages or running stability tests gives me the shivers. I bow to you and your superior geek cred, oh mighty overclocker.

What do you do for cooling?

I have a 14000 BTU/hr portable air conditioner that is more than able to keep up with the load. I asked my contractor to install an exhaust vent in the wall of the server room leading outside (like you'd use for a clothes dryer), allowing the A/C to exhaust hot air.

My house does not actually have any central air conditioning. Only the server room is cooled. We only get a couple of uncomfortably-hot days a year around here.

Dragging over your own computers is part of the fun of LAN parties. Why build them in?

I know what you mean, having hosted and attended dozens of LAN parties in the past. I intentionally designed the stations such that guests could bring their own system and hook it up to my monitor and peripherals if they'd like. In practice, no one does this. The only time it ever happened is when two of the stations weren't yet wired up to their respective computers, and thus it made sense for a couple people to bust out their laptops. Ever since then, while people commonly bring laptops, they never take them out of their bags. It's just so much more convenient to use my machines.

This is even despite the fact that up until now, my machines have been running Linux, with a host of annoying bugs.

How did you make the cabinetry? Can you provide blueprints?

I designed the game stations in Google Sketchup and then asked a cabinet maker to build them. I just gave him a screenshot and rough dimensions. He built a mock first, and we iterated on it to try to get the measurements right.

I do not have any blueprints, but there's really not much to these beyond what you see in the images. They're just some wood panels with hinges. The desk is 28" high and 21" deep, and each station is 30" wide, but you may prefer different dimensions based on your preferences, the space you have available, and the dimensions of the monitor you intend to use.

The only tricky part is the track mounts for the monitors, which came from ErgoMart. The mount was called "EGT LT V-Slide MPI" on the invoice, and the track was called "EGT LT TRACK-39-104-STD". I'm not sure if I'd necessarily recommend the mount, as it is kind of difficult to reach the knob that you must turn in order to be able to loosen the monitor so that it can be raised or lowered. They are not convenient by any means, and my friends often make me move the monitors because they can't figure it out. But my contractor and I couldn't find anything else that did the job. ErgoMart has some deeper mounts that would probably be easier to manipulate, at the expense of making the cabinets deeper (taking more space), which I didn't want to do.

Note that the vertical separators between the game stations snap out in order to access wiring behind them.

Here is Christina demonstrating how the stations fold out!

What games do you play?

Off the top of my head, recent LAN parties have involved Starcraft 2, Left 4 Dead 2, Team Fortress 2, UT2k4, Altitude, Hoard, GTA2, Alien Swarm, Duke Nukem 3D (yes, the old one), Quake (yes, the original), and Soldat. We like to try new things, so I try to have a few new games available at each party.

What about League of Legends?

We haven't played that because it doesn't work under WINE (unless you manually compile it with a certain patch). I didn't mind this so much as I personally really don't like this game or most DotA-like games. Yes, I've given it a chance (at other people's LAN parties), but it didn't work for me. To each their own, and all that. But now that the machines are running Windows, I expect this game will start getting some play-time, as many of my friends are big fans.

Do you display anything on the monitors when they're not in use?

I'd like to, but haven't worked out how yet. The systems are only on during LAN parties, since I don't want to be burning the electricity or running the A/C 24/7. When a system is not in use during a LAN party, it will be displaying Electric Sheep, a beautiful screensaver. But outside of LAN parties, no.

UPDATE: When I say I "haven't worked out how yet," I mean "I haven't thought about it yet," not "I can't figure out a way to do it." It seems like everyone wants to tell me how to do this. Thanks for the suggestions, guys, but I can figure it out! :)

The style is way too sterile. It looks like a commercial environment. You should have used darker wood / more decoration.

I happen to very much like the style, especially the light-colored wood. To each their own.

How much did all this cost?

I'd rather not get into the cost of the house as a whole, because it's entirely a function of the location. Palo Alto is expensive, whether you are buying or building. I will say that my 1426-square-foot house is relatively small for the area and hence my house is not very expensive relative to the rest of Palo Alto (if it looks big, it's because it is well-designed). The house across the street recently sold for a lot more than I paid to build mine. Despite the "below average" cost, though, I was just barely able to afford it. (See the backstory.)

I will say that the LAN-party-specific modifications cost a total of about $40,000. This includes parts for 12 game machines and one server (including annoyingly-expensive rack-mount cases), 12 keyboards, 12 mice, 12 monitors, 12 35' HDMI cables, 12 32' USB cables, rack-mount hardware, network equipment, network cables, and the custom cabinetry housing the fold-out stations. The last bit was the biggest single chunk: the cabinetry cost about $18,000.

This could all be made a lot cheaper in a number of ways. The cabinetry could be made with lower-grade materials -- particle board instead of solid wood. Or maybe a simpler design could have used less material in the first place. Using generic tower cases on a generic shelf could have saved a good $4k over rack-mounting. I could have had 8 stations instead of 12 -- this would still be great for most games, especially Left 4 Dead. I could have had some of the stations be bring-your-own-computer while others had back-room machines, to reduce the number of machines I needed to buy. I could have used cheaper server hardware -- it really doesn't need to be a Xeon with ECC RAM.

Is that Gabe Newell sitting on the couch?

No, that's my friend Nick. But if Gabe Newell wants to come to a LAN party, he is totally invited!

UPDATE: More questions

Do the 35-foot HDMI and 32-foot USB cables add any latency to the setup?

I suppose, given that electricity propagates through typical wires at about 2/3 the speed of light, that my 67 feet of cabling (round trip) add about 100ns of latency. This is several orders of magnitude away from anything that any human could perceive.

A much larger potential source of latency (that wouldn't be present in a normal setup) is the two hubs between the peripherals and the computer -- the powered 4-port to which the peripherals connect, and the repeater in the extension cable that is effectively a 1-port hub. According to the USB spec (if I understand it correctly), these hubs cannot be adding more than a microsecond of latency, still many orders of magnitude less than what could be perceived by a human.

Both of these are dwarfed the video latency. The monitors have a 2ms response time (in other words, 2000x the latency of the USB hubs). 2ms is considered extremely fast response time for a monitor, though. In fact, it's so fast it doesn't even make sense -- at 60fps, the monitor is only displaying a new frame every 17ms anyway.

Do you use high-end gaming peripherals? SteelSeries? Razer?

Oh god no. Those things are placebos -- the performance differences they advertise are far too small for any human to perceive. I use the cheapest-ass keyboards I could find ($12 Logitech) and the Logitech MX518 mouse. Of course, guests are welcome to bring their own keyboard and mouse and just plug them into the hub.

Why not use thin clients and one beefy server / blades / hypervisor VMs / [insert your favorite datacenter-oriented technology]?

Err. We're trying to run games here. These are extremely resource-hungry pieces of software that require direct access to dedicated graphics hardware. They don't make VM solutions for this sort of scenario, and if they did, you wouldn't be able to find hardware powerful enough to run multiple instances of a modern game on one system. Each player really does need a dedicated, well-equipped machine.

I'll keep adding more questions here as they come up.

79 comments:

  1. Nice Ken!
    Which brand do you use for the 2U and 3U cases?

    ReplyDelete
  2. Great info!!!! Hey are you trying to hide the actual "server" room?? please, post a couple of more photos of that part, all the "backbone", "guts" of your house! Thanks for all the info!!!

    ReplyDelete
  3. Thanks for the post! More pictures would be welcome, if you don't mind. (Just the techie stuff) :)

    ReplyDelete
  4. Still a bit confused on the whole netbook setup. Once I move out and get enough money to set this up, I will probably have a few more questions. But thanks for all the information.

    ReplyDelete
  5. could you tell me which mounts / pole system you used for the LCD's ? I would love to make something similar for my living room on a much smaller scale. (maybe 1-2 machines)

    ReplyDelete
  6. I love the use of Ubuntu for this! I manage a bunch of elementary school labs using this exact method for booting their diskless terminals. Could you go into some more detail on the Win7 install on iSCSI with an LVM backend? I'd love to give this a try myself.

    ReplyDelete
  7. I've added more pictures to this post!

    @Jerry: They are a brand called Athena Power, but I wouldn't necessarily recommend them. The options for rack-mount cases were surprisingly bad. There was one from Antec that looked nice but it was overly expensive.

    @Steve G: I added some more info about these to the FAQ about the cabinetry.

    @Shujin: I linked to instructions at etherboot.org on how to install. Just follow them. It's quite easy -- Windows 7's installer simply lets you select the iSCSI drive as the install disk, so long as you have properly bootstrapped using gPXE.

    ReplyDelete
  8. Your house is awesome. Thanks for the technical details!

    ReplyDelete
  9. The copy-on-write SAN setup really intrigues me. It seems like something like that would be pretty ideal for a work environment.

    ReplyDelete
  10. @Kenton - Great write-up. Thank you for all the pictures & details. I think I'm going to attempt to do something similar, albeit on a smaller scale.

    ReplyDelete
  11. @Kenton - First of all, AMAZING. I've been crunching numbers, and trying to figure out how to convince my wife to buy off on my upgrade!

    For iscsi boot with cow, You may want to try http://www.iscsicake.com We looked at it a while ago to iscsi boot lab workstations, and it seems to be pretty stable. Uses COW and local client-side storage for cache as well.

    Downside is it's windows OS based, but it will run off a client windows OS. Just a thought.

    ReplyDelete
  12. Love the setup - however, there's three areas that you could have cut costs: The rack mount, the cases and the cabinetry. Both have the same problems involved, however: It's not the quality of the wood I am reffering to. Rather, it's the time spent into it - particularly, you doing it yourself. Because of how simple the design looked, you might have been able to knock off at least a third of the 18k investment As for the cases and the rack mount, because of how you have it set up, you might have been able to get away with building a setup which would have saved space, kept things cool, been easily upradable, and - most importantly, knock off half-two thirds of the cost. The problems for all of this would be the same no matter which you did: Time it would take to do it, dust and tool considerations for if you don't have the stuff to do it, and your own skill level. Because of how simple it all looks, it probably wouldn't have taken long in either setup, even as a beginner.

    Still, the setup is freaking awesome - thanks for sharing the info on how it works! :D

    ReplyDelete
  13. The problem with Sonic (as with all DSL) is their upstream bandwidth is horrid. I pay ~$50/month for Comcast and have 7Mbps upstream.

    ReplyDelete
  14. As a geek who is also a woodworker, I'm confused when you say you used solid wood. The cabinets looks like they're made of plywood. Is that correct? If you used solid wood to build the entire thing, I think you may have some pretty severe warping / expansion / contraction issues in the next couple years as solid wood is not dimensionally stable like plywood is. Btw, is that maple or some other species? It looks nice.

    ReplyDelete
  15. @jjcrandall: Yes, I've seen iSCSI Cake, but I shied away from it because it is proprietary, costs money, and would require a Windows server. The first point is the most important -- if iSCSI Cake did not quite work for me for some reason, I'd be SOL. I'd rather have something that I can take apart if I have to. (Of course, same goes for Windows itself, which is why I tried to use Linux at first. But at least I pretty much know what to expect from Microsoft, and know there are lots of resources on the web to help me with it.)

    @w2ed: Indeed, I could have saved money by doing more things myself. I don't know any woodworking, though, so I decided it made the most sense to out-source that.

    @Fred Weston: As a geek who knows nothing about woodworking, I'm sure I don't actually know what I'm talking about and you're probably right that it is plywood with a maple veneer. (It is maple, though.)

    ReplyDelete
  16. Thanks for posting all the details, I'm not a gamer but I like all the technical details. Nice to see something new in the geekhaus design pool.

    ReplyDelete
  17. Thanks Kenton for the mention, appreciate your support!

    -Dane

    ReplyDelete
  18. As another geek with woodworking tendencies, I've found a lot of contractors refer to plywood as "solid wood" and MDF as "particle board". No reputable carpenter would use actual plain wood for something like this. It'd warp in a year tops and you'd put in a claim against their bond for not doing it in a workmanlike manner. To those not woodworking-inclined, using quality plywood is actually better than not. It's the quality that matters, really.

    Great writeups, Kenton. I've been following them on Slashdot and such with interest. As an independent IT guy, I find it highly interesting for possible use in my own home office someday.

    ReplyDelete
  19. This comment has been removed by the author.

    ReplyDelete
  20. I wish I lived near you just so I could get service though Sonic.net. Though, for the first time in years, I'm currently living in place where I haven't had a major internet related problem not caused by my own hardware (had a few bad routers). TimeWarner cable internet is surprisingly steady.

    Also, nice write up. One day, if I ever own a house, I'm going to do something sort of like this.

    ReplyDelete
  21. You mentioned using gPXE. gPXE has been superseded by iPXE. http://ipxe.org/

    ReplyDelete
  22. @Chaz6: Thanks for the tip. Somehow in all my Googling I didn't find that.

    ReplyDelete
  23. With regards to using the displays when the host PC's are not in use, A HDMI Matrix Switch will let you easily put whatever you want on them without needing the host PC's to be turned on. However with the number of sources and sinks you would need a switch, a 16x16 switch will not come cheap.

    ReplyDelete
  24. This comment has been removed by the author.

    ReplyDelete
  25. You could use a couple of these:

    http://goo.gl/yQ0ga

    Would allow you to put any input anywhere, but at $6.5k each, you could buy an awful lot of electricity to run the computers instead...

    ReplyDelete
  26. Did you look at LTSP (linux terminal server project) when you set up your linux systems? It's really easy to network boot the client machines, and only about four or five commands on the server to install that end. Best way to demo it is download a copy of edubuntu.org and burn it to disk Boot up the master node and then your clients to try it out.

    Also look up the Linux Gamers live DVD.

    ReplyDelete
  27. @Kenton

    Thanks for the response and like most i bet, I appreciate you diving into the details of your project with the two follow up posts. What you stated makes perfect sense.

    What I really like your overall design. The quality of the cabinets is amazing, and even in the server room there's been a great attention to detail.

    Where did you get your server rack?

    ReplyDelete
  28. If you use something like openIndiana you can export your storage volumes as iSCSI targets directly. With this you can use ZFS deduplication and SSD caching.

    http://blog.zorinaq.com/?e=41

    Another option in the future may be a product called BeTwin VS. A user on the HardForums had a bit of success with this. Basically it turns a Windows 7 installation into a multi-seat box. You could take a system with four video cards and have four machines.

    http://hardforum.com/showthread.php?t=1598195&page=4
    (Yes I know HardOCP is kind of lame but that particular post was pretty good.)

    ReplyDelete
  29. @jjcrandall: Got the server rack from Amazon. :) It's just some Tripp Lite 4-post thing. Nothing special.

    @LC: I am exporting my storage volumes as iSCSI directly. :) ZFS dedup could be nice but there's no ZFS on Linux due to petty licensing disputes. :( Maybe btrfs will catch up someday.

    BeTwin VS wouldn't work for my setup because these are gaming machines. Games are resource-hungry so running four instances on one machine with four video cards would really not work. Would be great for a setting where people just needed a web browser, though.

    ReplyDelete
  30. @jvin248: Since these are game machines, a "thin client" solution really isn't going to work. Games are resource-hungry beasts that need a well-provisioned system and direct access to hardware to run smoothly.

    ReplyDelete
  31. There are actually about three ZFS-for-Linux projects, but I wouldn't run it on Linux. openIndiana is FOSS and easy to use.

    BeTwin VS works for games. Check out the Hardforum thread. In a lot of situations the cost delta for a bigger single box (SR-2, SR-X, etc) with another video card is less than the cost of two smaller separate boxes.

    ReplyDelete
  32. > Is your electricity bill enormous?

    > I typically have one LAN party per month. I use about 500-600 kWh per month, for a bill of $70-$80. Doesn't seem so bad to me.

    how?!

    I have a 1200sq.ft. house and per month apparently the it consumes 1200-1800kwh per month.. I have approx 1/4th of the power suckers in the house.

    curious what you do to regulate power consumption?

    ReplyDelete
  33. i think i have idea or two how too ran the screens in of mode with just one or two computers active. but that depends on what you want to do.

    all the screens displaying the same view ?
    or each different one ?
    or even groups ?

    ReplyDelete
  34. @Doctor Partlow: I only turn the game machines on for around 12-24 hours per month. If each one takes 300W-500W, then the total monthly electricity usage of my gaming machines is somewhere between 43kWh and 144kWh. Even on the high end, the gaming machines are less than a quarter of my total electricity usage, and my house does not even use that much electricity (no electric heating or cooling, plenty of natural lighting, etc.).

    ReplyDelete
  35. To be able to display photos on your monitors put one of these in place so that when the LAN computers are on it switches to that input, otherwise it switches to the photo input:

    http://www.monoprice.com/products/product.asp?c_id=101&cp_id=10110&cs_id=1011001&p_id=8148&seq=1&format=2

    Get Roku 2 HD for each station (or the $49 LT if 720p is good enough) and install the Picasa channel. Run them in slideshow mode 24/7. Or have up one specific picture or work of art.

    Then whenever not in LAN mode, the switch will bring up the Roku 2 in slideshow mode. When the computers boot, it will automatically switch to the LAN computer.

    Roku 2's take very little power and then the only additional power would be for the monitors which would be a good trade-off for having your photos or great works of art displayed when you want.

    ReplyDelete
  36. I'm guessing that you went with the 3U rackmounts for noise, cost, and cooling?

    I would have figured that with having 12 sytems, you probably could have done with just a single 12-core (dual Xeon X5690) system in a 2U rackmount and then partition that up into VM hypervisors?

    I'm also guessing that blades would have costed you more, but you probably would have saved in space, power, and cooling in the end.

    Neat though.

    ReplyDelete
  37. How would you get a dedicated accelerated video output to each of the 12 systems that way?

    ReplyDelete
  38. @alpha754293: No, you can't run games in a VM. They need dedicated graphics hardware. Not to mention, multiple CPU cores, high memory bandwidth, etc. I went with 3U cases because that's the smallest kind that can fit a high-end video card.

    @Andrew Schultz: Thanks for the tips but I think I can figure out what to do with the screens when I get around to it. :)

    ReplyDelete
  39. @Kenton Varda

    I have no doubt, just another datapoint along the way. Enjoyed the writeup!

    ReplyDelete
  40. Hi Kenton,

    Wow I'm impressed with your ingenuity. I run a lab of 32 machines (no hd's required) using DRBL (Diskless Remote Boot Linux). It also uses pxe to boot but uses nfs for the mounts. Or you could simply use Clonezilla to multicast image the client hd with Win7.

    ReplyDelete
  41. Hi Kenton you have done an amazing job! Well Done!

    I would like to learn more about the technical aspects specifically the PXE booting. I didn't know that TFTP could support multiple dynamic hosts reading from the same "distribution point".

    Could you point me to some information that helps overcome the need for TFTP static directory name exports based on hostname/ip addresses for the kernel/initrd images?

    ReplyDelete
  42. DRAiNO: I assume you're asking about the original Linux-based setup since you're asking about initrd. In that setup, I actually serve exactly the same data to all the clients, since they set up their copy-on-write overlay on the client side. However, when pxelinux boots, it first looks for a config file at a path based on the client machine's IP address, before falling back to the default config. Therefore, it is easy to tell it to do different things for each machine. TFTP itself does not need to know the difference.

    Note that TFTP is only used to download pxelinux, pxelinux's config, the linux kernel, and initrd. After that, everything is done through NBD and higher-level protocols rather than TFTP.

    In the Windows-client setup, TFTP is only used to download gPXE, which in turn connects directly to iSCSI. In this setup, gPXE decides which iSCSI target to connect to based on parameters sent from the DHCP server. The server is configured to send different parameters to each client based on MAC address.

    ReplyDelete
  43. "The monitors have a 2ms response time (in other words, 2000x the latency of the USB hubs). 2ms is considered extremely fast response time for a monitor, though. In fact, it's so fast it doesn't even make sense -- at 60fps, the monitor is only displaying a new frame every 17ms anyway."

    That would be true, if 2ms were the worst-case response time. The spec posted on the monitors is gray-to-gray, which is close to best-case. The black-white-black numbers, which measure the most extreme switch the monitor can do, are often much higher; well above 17ms with many of the "5ms" and "8ms" displays. Gamers tend to advocate the "2ms" panels because even their slowest transitions are generally faster than 17ms, which avoids the smearing and ghosting issues that are so reviled on slower LCDs.

    ReplyDelete
  44. In response to "Why not use thin clients and one beefy server / blades / hypervisor VMs / [insert your favorite datacenter-oriented technology]?"

    Xen and KVM both have PCI pass through, but there's something else with the bios interrupts that is required in addition to this for Video cards. Xen has some patches applied already to this effect. So between Hardware Virtualization for the CPU and PCI passthrough to the Video cards, I would imagine that you could get at least 4 clients on an 8 core, 4 GPU, 64GB machine running games just fine. This is all theoretical ofcourse, but I would love to see somebody with more spare time (and hardware) than I to test this out. I'm still a few years out from my planned "Ultra Virtual Home Computing" setup (oldest kid is 4).

    ReplyDelete
  45. @Kai Meyer: Find me a motherboard that has two CPU sockets and four PCIe x16 slots. :) Also remember that even if you can cram all that hardware in there, it's likely still sharing a single bus and so will be bandwidth-starved.

    ReplyDelete
    Replies
    1. Coming soon from SuperMicro: X9DRG-QF

      4 x16 PCIe-3.0 w/ double spacing for GPUs. If it's like their other 4 slot solution the bus is dedicated 2 for CPU1 and 2 for CPU2.

      Perhaps a step closer to virtualized game boxes?

      Delete
  46. I'm not a hardware enthusiast by any means. I have pipe dreams that when I have the motivation, there will be hardware to do what I want :) I love your setup, and would duplicate it in a heart beat given the means. If I could wave a magic wand and make any change, I would swap Virtual Machines for iSCSI. Given the (apparent) constraints, I love what you've done!

    ReplyDelete
  47. Kenton,
    Maybe I'm missing something...but would you mind listing all the software you used? I am more than willing to figure out how to make it all work, but I'm having a heck of a time finding all the software. I know you said it's open source, but I don't even know what to search for. Thanks.

    ReplyDelete
  48. @David Hatch - I used Ubuntu. There were a bunch of packages I installed at various times, and I don't think I could give you a complete list, but everything came from the Ubuntu distribution. E.g. if you search for "ubuntu dhcp server" you'll get info on the DHCP server I used (I guess it's the ISC server). Now that I think of it, I had to use tftpd-hpa rather than the default tftpd package in order to support PXE properly, but both packages are available from Ubuntu.

    ReplyDelete
  49. Ok, well maybe I don't know what I'm trying to search for. I have a DHCP server already on the network so I will just continue to use that I think. As for the rest, if you don't know the names, that's fine but as far as I can tell you have a TFTP server, some iscsi client, a COW client/utility, and...is that it? I guess I'm just trying to figure out which pieces I need. I work for some schools and this would be WONDERFUL to put into production there. I understand that you're busy and I appreciate what you have shared already. Thanks.

    ReplyDelete
  50. As an aside, I am familiar somewhat with PXE booting as I currently use Windows Deployment Services to deploy my Windows images across our network, but something like this I can see being even more useful on many other levels.

    ReplyDelete
  51. @David Hatch - The COW support is based on LVM snapshots. LVM is Linux Volume Manager, a standard part of Linux. The iSCSI server is whatever one is recommended for Ubuntu. The iSCSI client is Windows 7 -- it's included with the OS (search for "Microsoft iSCSI initiator"). You will need gPXE in order to help Windows boot from iSCSI, as described in the article at etherboot.org that I linked above -- I believe this is the _only_ piece of the puzzle that isn't part of Ubuntu or Windows. Otherwise I think that's it.

    ReplyDelete
  52. Sounds good. Appreciate the info. I've always wanted to dabble a little more in Linux, so this should help get me started. Thanks again for the great ideas! Hopefully this will help me at my school, especially with the computer labs where students break stuff constantly.

    ReplyDelete
  53. The project is very enovetive.The hardware elements configaretion also sound.

    ReplyDelete
  54. you ever think of making the monitors automatically slide down ?

    ReplyDelete
  55. If I wanted to implement something similar to this using Windows Server and either iSCSI SAN hardware or a software based iSCSI solution for the server side, what would you recommend?

    ReplyDelete
  56. Also, this setup is awesome. I now feel like looking into blade systems to see if they are a viable alternative to building individual machines. I'm fairly certain that blade workstation solutions exist that have the horsepower needed to run modern games quite well, it's just going to be a question of price compared to purchasing/assembling components.

    ReplyDelete
  57. @fonsui: I don't know a whole lot about iSCSI SAN hardware, but I'd recommend against it. I am using a software-only solution and it works fine. Specialized hardware tends to be expensive -- and specialized, meaning not flexible. It's probably easier to solve performance problems by adding more servers on independent subnets.

    I don't know a whole lot about blades, but it seems like blade workstations are designed under the assumption that they live in a data center far away from the user. For gaming, you're going to want to string an HDMI or other video cable directly from the machine to the monitor. If you have to send video over the network, it's going to end up significantly degraded and you'll probably incur latency. Plus blades are expensive and specialized. Upgrading may get tricky.

    In general, as cool as all this specialized hardware may look, I think sticking with commodity hardware is going to be more successful in the long run.

    ReplyDelete
  58. The particular blade system I was looking for can utilize PCIe display adapters, which would allow me to connect my displays directly to the machines, since you're quite right about the degraded performance that would be experienced if video output was sent via network. The important point though is that the chassis (Dell M1000e) for these blades (Dell M610x), and the blades themselves, end up making the price unreasonably high for such a project. The individual blades cost around $2k with the most basic config, while the chassis and additional components would run probably around $10k or even more. Add on top of that the cost of the display adapters themselves, and that's assuming there are no issues with using non-recommended hardware and that they'll work. As sweet as a blade system would be, consumer hardware it is.

    As far as the image management and storage is concerned, do you have any ideas as to what software package(s) might fit the bill if I wanted to use Windows Server as my back end as opposed to Linux?

    ReplyDelete
  59. Yeah in my case I only spent $1000 per game machine (not including monitor, but including expensive rack-mount chases), and these machines can pretty much run any popular game at max graphics settings. So if you spend more than that, you're spending too much. Also, with commodity parts, you can upgrade incrementally, so maintenance costs are much lower.

    I don't know much about Windows Server, which is why I used Linux. :) But I think:
    - An iSCSI initiator is built into the OS.
    - The DHCP server obviously comes from Microsoft.
    - You may still need gPXE in order to boot over iSCSI, but that's OS-independent.
    - To get copy-on-write overlays, you may need to use VHDs instead of physical disk partitions. I believe you can create a VHD which is a COW overlay over another VHD, and you can probably export VHDs on iSCSI. But I don't know the details.

    Alternatively, if you're willing to spend money on proprietary software, I believe there are commercial solutions that implement SAN master + local COW overlay. I think one of them is called CCBoot, but there are others as well.

    ReplyDelete
  60. by the way what sort of wood did you use

    ReplyDelete
  61. @Unknown: Maple veneer on plywood, I believe. I didn't personally build the things, though.

    ReplyDelete
  62. Kenton, this is awesome stuff. Thanks so much for sharing. Hoping to follow your instructions to a similar setup with Windows 8 as the base OS.

    One preliminary question that I'm wondering if you've had any experience with: does EA Origin tolerate the single install, multiple user account scenario like Steam does? Would love to use Battlefield 3 in LAN parties, but it's unclear whether it's feasible. This link implies that it is somewhat possible but I'm wondering whether you play this game and might know better:

    http://forums.electronicarts.co.uk/battlefield-3-pc/1449812-multiple-origin-accounts-bf3-same-computer.html

    Thanks again for sharing.

    ReplyDelete
  63. @Matthew S. Marino: Good question. To be honest, I had so much trouble getting Battlefield 3 to work at a "normal" LAN party where everyone brought their own computers that I never bothered trying it in my house. I probably won't touch EA Origin again for some time.

    https://plus.google.com/118187272963262049674/posts/bBF7sCeWym4

    ReplyDelete
  64. When I first spotted this post several months ago, I thought it was the coolest idea ever. But now I've thought of something and I wanted to post here about it.

    I'm a part owner of an internet cafe/gaming center, where we have about 30 PCs. The idea of locking away all the PCs in a single room that doesn't have random customer passing through it, and has it's own cooling fascinates me. Do you suppose this design would scale up to 30-50 stations?

    The benefits would be the added security (only thing the customer has physical access to is the monitor, keyboard, mouse, and headset), plus we wouldn't have to run nearly as much cooling due to the PCs and the customers being in one area. What are your thoughts on this? Do you think this would be feasible in a commercial setup?

    ReplyDelete
  65. @Phaux: If it's just keeping the computers in a separate room that you care about, that's easy. All you need is some long USB and HDMI cables -- get them at monoprice.com. There's no special software required and it scales to as many machines as you have space for. :)

    The netboot setup is another thing entirely (and is independent of the physical location of the computers). If you only have one server and one network link, each additional machine will slow things down a bit. I suspect my 12 machines are pushing the limits somewhat. However, you can always add more servers, or perhaps even just add more network adapters to the server, to distribute the load. With multiple servers, if you still wanted to only have to install updates once (rather than once per server), you would either need to clone the disk image between servers after any update, or you would need to have a master/slave setup where you have a bunch of servers that are basically proxies for a master server. The latter approach could actually work pretty well if everyone typically loads the same content at the same time since each proxy would cache the content and so would only request it from the master once. But, it would be more complicated to set up, and latency could be an issue for the first person who loads a game.

    ReplyDelete
  66. @Kenton: Yeah, the software side is easy, and already figured out. The part I was worried about was the cooling demands of 50 PCs in a small room. The other part was how big said room would need to be, and am I right thinking that I'll need 4 full racks to fit all 50 PCs + Server(s) + Network Devices? Assuming the rack is 42U tall and each PC is 3U.

    ReplyDelete
  67. @Phaux: Cooling is just a matter of calculating (or measuring) the total wattage of all the machines and then buying enough air conditioning to match. Most air conditioners are measured in BTU/hour, which is just an alternative unit for watts -- 1 kW = 3412 BTU/hr. My 14000 BTU/hr AC has no trouble keeping up with 12 PCs.

    As for space, yes, four racks sounds right.

    ReplyDelete
  68. Ken, if i'm not mistaken (from ur pic above), I think you better position the AC "in front" of your rack. Because (AFAIK) the fans of your machines suck the air from front-side and blow it behind.

    ReplyDelete
  69. @nookie:

    1) The AC is, in fact, "in front" of the rack. The fans you see on the machines in the picture are intake fans, on the front.

    2) As you can see in the pictures, there is plenty of space for air to circulate around the rack. Moreover, the heat risk is a slow, gradual one -- if I turned off the AC with the machines still running, it would take tens of minutes, maybe hours for the room to heat up to the point of causing problems. So, there is plenty of time for air to circulate. As such, it doesn't matter where the heat is extracted from the room. All that matters is that it is extracted at a sufficient rate to keep up with the computers introducing new heat.

    3) My name is Kenton, not Ken.

    ReplyDelete
  70. Hi Kenton Varda, I do like what you did and thanks
    to share your project.
    But, for the new setup (Windows7) I'd like to know
    how you implement the COW overlay server side so that
    every machines can boot up to the network using the
    same Windows 7. I'll really apreciate if you can explain
    the technical side of that to me because I'd like to do
    it myself.

    ReplyDelete
  71. @lee aurele: As I said in the article, I used LVM snapshots. The master image is a 400GB partition, and from that I have created 12 LVM snapshots, each 20GB in size. Each machine connects to a different snapshot.

    There is lots and lots of documentation out there on how to use LVM; I don't think I could add anything useful on top of that.

    ReplyDelete
  72. @Kenton: Thank you.
    So according to what you said, at the end of day you have 12 iscsi targets.

    ReplyDelete
  73. @lee aurele: Yes, exactly. Or possibly 13, if you want to be able to connect to the main image when performing updates. (Though, I tend to do updates in a snapshot, and then do lvconvert --merge to merge them back into the master afterwards.)

    ReplyDelete
  74. @Kenton: Ok and once again Thanks.

    ReplyDelete
  75. @NormanFor8: Even if I hadn't deleted your SEO linkspam comment, it would not have done you any good. All links in Blogger comments automatically get annotated with rel='nofollow' which means Google will ignore them. You are wasting your time.

    ReplyDelete