Keith Demele's Blog

So, while I was sitting in my biology class Tuesday night, I checked my email and noticed I had one from someone I emailed on craigslist a while back. This guy had a 1U server on craigslist for pretty cheap, however, at the time I didnt want to buy another machine for the farm, this was around the time I was doing my first build…Anyway, he needed cash and was willing to give it to me half price if I picked it up that night…Really good deal, so after class I drove an hour over to fort worth, met him, and picked it up…Its a 1U 3.2ghz Celeron with 2gb of ram, and a 80gb HD, and I got it for less than what the case for it costs retail, and I should be able to upgrade it in the future…It runs strong, I put a webserver on it, for tesing purposes, and I also put backburner on it, so the machine will be my dedicated rendering job manager. Basically it tells the other machines on the network what to render when, to get stuff done in a timely manner. The hardest part of the whole deal so far has been networking, and permissions, there are a lot of config files I have had to tweak…but now I can render on the render machines and they will put the files on my workstation, and all the computers recongize eachother…so thats good…

I will either be adding another Xeon machine or maybe a storage server in the future, we will see. The downside to only having 3 machines rendering, (the two in the rack, and my desktop), is that even though its fast they can only do so much. For example, if I had 10 dual core machines, it would take twice as long to render a frame on each machine, but since there were 10 of them they got that much more work done. Oh well, I dont think it was feasable to go with older dualcores now, it would only be $200-$300 cheaper per machine, and it would be old technology.

Anyway, heres the machine:

And here it is in the rack all hooked up.

So two weeks ago I got all my stuff in for my 2nd renderfarm box, I put it together and got it running but neglected to post on here for some reason, now that I am home from work and need to study for a statistics test in the morning I figured I would fill you all in, and procrastinate a bit…

Here it the farm so far:

The build:
iStarUSA Black 2U Rackmount Case (350w)
Intel X3220 Xeon 2.4ghz
Gigabyte GA-EP35-DS3L Mobo (EP= Energy saver/green)
80gb Western Digital
8gb Ocz Gold ram

So, the build went pretty fast but I had a lot of trouble overclocking this thing, but more on that later…I went with a Xeon because I thought the price trade off from a Q series chip was well worth the 15c variance in thermal specifications, that is, the Xeon can take hotter temperatures (turned out coming in VERY handy). The chip is just like the Q6600 in other respects, as far as you need to know. On to the build…So, the 2U case was, well, 2U, thats 3.5″ tall, which makes things interesting, you have to use a power supply and you have to find a heatsink which is small enough to fit in there.

So, here is a pic before we get down to the nitty-gritty:

As you can see I needed to fit all of those parts into that small case, small compared to a standard case, that is. Yes it does fit, and no the router went in the rack, not the case…

So after I got everything from the post office, I came back home and put it all together:

Now, because this is sitting in a rack and I have no need for opengl or any type of graphical rendering I figured that all I normally needed was a remote desktop connection, so, there was no need for a video card. However, you do need a video card to install the OS, overclock, and while you are configuring the network settings…So, before I booted the machine for the first time I had to put one in (I have a few spare cards laying around), sp when I first tried to put a video card in, I used the little riser they have for the PCI slots so they will work with the case (notice the slots in the case are sideways instead of vertical), so I put the card in, it fit fine, and I proceeded to boot the machine. In a blink of an eye my room was full of black smoke! Before I could turn off the machine, a few of the circuits in the extender totally melted down…Thats no good…

Since I could not put a video card in the case anymore, I had to pull the supports out of the case, then the tray, and prop it ontop of the machine…

Now, when I tried to flip the switch after I did this the machine would not boot, did I fry the mobo? No not quite, I just didnt have sufficient ground. I had the metal of the tray touch the case some more and it fired up fine. Once I booted I installed the OS, confuigured the network, updated drivers and bios, installed my programs, and proceeded to overclock, and here is where it got a little crazy. Error after error, after a week of testing and tweaking I finally got a stable 3.0 overclock. The hard drive corrupted, I had errors with DMI, the video got all wonky, and several other things went very, very wrong…

Anyway, I finally got it configured and running right. The power supply is by far the loudest thing in my room, over my 4 other computers and window AC unit, its like an F14 on the tarmac, the worthless heatsink doesnt help either…I may need to do some modding down the road…My load temps are around 65c well within the range of the chip, and the temp of the server rack is about 95F…After I got the machine up and running I put it in the rack with the new router (nice, but slow) and did some cable management…Here is the result…

And for anyone wondering, so far the rack pulls 330w at peak, and by my extremely loose estimate costs about 65 cents a day to run at 100% load.

Anyway, I should get back to cramming studying for that test…

Ahh random installment numbers, how fun…..Render Machine #1 is up and running in the rack now, running at 3ghz with 45C load temps, in my much coolder than cool room…I could not be happier with the results. Needless to say, It’s a rendering MACHINE! I can’t wait till I fill the rack with these badasses!

Few things I did learn was that the cords that come with the case’s PSU are way to short to do anything, I couldnt even reach to the far end of the case to run the power cord to the processor, so I had to make due…I also learned that I can pull the video card out after I am done with the configuration and it will drop my temps by 10C! I simply remote desktop in to check temps or to do something like that…So, after I took most of these pictures I pulled the video card and let it work away…Here are some pictures with some more info and process, bare with my extremely messy room, I had to take down some storage racks to make room for the cabinet, and I was so excited to get the system up I didn’t bother cleaning…lol, anyway, here we go…

(fan and Zalman 8000 not pictured)

I plugged the case in and it did nothing so I thought the PSU was dead on arrival, I simply jumped the PSU to test it, and found out its good to go, so time to move on…Mental note, and a bit of advice, a shock from the PSU SUCKS!!!!

it works:

After the PSU was tested I needed to install the mounts for the mobo, always be sure that these mounts match the holes on the mobo, you feel stupid when one or two are in the wrong place:

The insulators you see in red there are not really required, but I put them there anyway, and before I mounted the mobo I had to install the heatsink mount:

Quad core power:

Here you can see the processor in socket, and the power cord for the processor stretched over the the port…

And here the heatsink, ram, and video card are in:

Then I hooked up the hard drive and all the rest of the junk, yeah I know, messy cables, they will be/were cleaned after everything was oiled and running smoothly….

It booted, time to install the OS (monitor on right):

Ready for a stress test over night, pre-OC:

So, I woke up the next morning in a SWEAT, it must have been about 90 degrees in my room, both my desktop and new machine were on the verge of overheating. What to do?

So, now that its a brisk 68 degrees ambient in my room, we can get to work…remember the power cord issue?

(its only temporary till I decide I am not going to RMA the case), then I will cut it…After updating the bios I overclocked it to 3ghz, nothing crazy, just enough to keep temps in range, but enough to make it worth it…stress tested again and set up networking, then put it in the rack:

Notice the thing dangling? Thats right its a thermomiter…

80 degrees in there with both doors closed…I didn’t have any double sided tape at the time, I will clean it up…

So, that about does it, its running stable at 45C load temps…

And, in case anyone was wondering, here is a quick rundown of my costs so far:

• Cabinet: $150
• Window AC unit $140
• 25′ Cat 6 cable: $12
• Case:$150
• Q6600 $190
• 4gb Ram: $100
• 7100gs: $20
• 80gb hd: $35
• Mobo $75
• DVD drive: $20
• Zalman 8000 $35
• temp gauges $20
• Software $150
• Shipping/gas $45

$1150 give or take, the next machine will be $650 or so, and my only other expenses will be a switch and a power strip, the switch I listed earlier fell through…Anyway, all and all a success I think…Unfortunately I will NOT be able to finnish this build/rack (ie add more machines) untill a but later down the road, life called, it wants my money, I mean, its money back.

So, I have gotten everything but my processor for my first case in the mail today, er yesterday…I only ordered the first case so I could make sure it fit in my audio rack…Anyway, after a quick diversion to frys, I got home and put the case into the rack to check fitment, low and behold it fits!

kinda:

In order for me to be able to close the rear door I will have to cut a hole in the side of the case and run my power through that, actually, I may just run the cord through one of the expansion slots and get all ghetto up in there…Issue #1 of 308468944 somewhat down, 308468943 issues to go.

More when I get the processor in today or tommorrow, damn UPS.

Well all the parts from render machine #1 have been shipped, as well as some extras for my workstation, and I should be putting together/testing the first render machine Monday or Tuesday at the earliest. A few hours after I ordered everything from newegg, the fine folks over at http://www.overclock.net/ showed me the build I had originally thought up, but could not find the parts too. It turns out that that build, which is 2U, is cheaper than my 3U build…The only downside is that the mobo can only take 4 gigs of ram…Regardless, next go-around I will probobly get that 2U…

Heres the build:
Athena Power 2U Rackmount – http://www.newegg.com/Product/Product.aspx?item=N82E16811192045
Intel Core 2 Quad Q6600 – http://www.newegg.com/Product/Product.aspx?Item=N82E16819115017
GIGABYTE GA-G31M-S2L – http://www.newegg.com/Product/Product.aspx?item=N82E16813128078
pqi TURBO 4GB DDR2800 – http://www.newegg.com/Product
EXCELSTOR Jupiter Series 80GB SATA – http://www.newegg.com/Product/Product.aspx?Item=N82E16822210003
ASUS Black 16X DVD-ROM – http://www.newegg.com/Product/Product.aspx?Item=N82E16827135151
/Product.aspx?item=N82E16820141366
Dynatron 77mm CPU Cooler – http://www.newegg.com/Product/Product.aspx?item=N82E16835114066

Anyways, again microcenter has the boxed processor cheaper, so I will save $20 or so more there…As you can see that mobo has integrated graphics saving me a bit…Will update you with Renderfarm Part 2.0.1 when I start assembly!

So upon further review, thought, and consolation I have decided that I will use 3U cases for my server rack. Newegg.com has one for sale with a power supply which should suit my needs, I am a bit hesitant to use that power supply, but its cheap and will do the job according to this nifty online calculator:

http://extreme.outervision.com/psucalculator.jsp

Here is the new build:

Athena Power Black 3U Rackmount Server Case –
http://www.newegg.com/Product/Product.aspx?item=N82E16811192028

GIGABYTE LGA 775 MOBO – http://www.newegg.com/Product/Product.aspx?item=N82E16813128084

Intel Core 2 Quad Q6600 – http://www.newegg.com/Product/Product.aspx?Item=N82E16819115017

OCZ Platinum (2 x 2GB) DDR2 1066 – http://www.newegg.com/Product/Product.aspx?Item=N82E16820227298

EXCELSTOR Jupiter Series 80GB SATA – http://www.newegg.com/Product/Product.aspx?Item=N82E16822210003

ASUS Black 16X DVD-ROM – http://www.newegg.com/Product/Product.aspx?Item=N82E16827135151

ZOTAC GeForce 7100GS – http://www.newegg.com/Product/Product.aspx?item=N82E16814500027

Grand total of $650 to my door…probably $600, I can find better deals on some of that here in town…

As for the case I have a few 120mm and 80mm fans I will install, something like this:

And here is a new flowchart:

Its starting to come together, I go pick up my gigabit switch either today or tomorrow. Found a good 16 port gigabit switch for cheap! Only deal is they lost the rack mount brackets, so I will have to fuigure something out…

Found this while I was doing research earlier…supposedly its one of Pixars research render farms, they have tons of rows of these apparently…one day:

So, by now you guys know that I am a 3d artist…Heres the thing,  I need a LOT of computing power to render out animated sequences in any decent amount of time. So, Instead of having a bunch of towers in my room I have decided to go rack mount….I went shopping and decided that a full 42U rack was way to huge for my room, and in general, I am in college and will find myself moving every year or so, so I decided to go with a smaller rack if I could find one, it would be more mobile, yada yada…Anyway, I saw and add for a 24U rack with plenty of depth on Craigslist and I went and bought it, I was in a rush to get back to work and forgot to bring a tape ruler, so I didn’t even bother looking at it to closely at his office. Stupid me when I got it home, I noticed that the rack only 15U, and after a few hours of research trying to fuigure out what brand it is, it turns out the case isn’t really a server rack, its a rack originally built for audio equipment. I have read that this will work, but I am concerned about heat management. if I go with it, there will deffinately have to be some airflow management modifications…Here’s a pic:

I guess my only options right now are to order my “renderblades” and hope they fit, if they don’t I can find a 42U rack and modify it to suit my needs I guess, but I love having casters on that thing, its so easy to move around. For anyone wondering, I guess a little outline to my project would help:

As you can see I will put my render nodes (the big cases in the rack) into a patch panel, which will then be put into a gigabit switch, (if I can get my hands on one….), from there I will control it from my workstation/node manager…that all that really needs to be said about the network/layout…I don’t claim to be an expert on this stuff so if anyone has any recommendations feel free to contact me.

Anyway, on to my render blades. My render blades need to be cheap, fast, quiet, and cool. That is a lot to ask for in a machine (x3+). I have devised a build which will give me this for about $680 shipped to my door, after rebates. This isn’t bad, here is the build:

OCZ StealthXStream 500W – http://www.newegg.com/Product/Product.aspx?Item=N82E16817341012
Norco 4U Case – http://www.newegg.com/Product/Product.aspx?Item=N82E16811219002
GIGABYTE GA-P35-S3G – http://www.newegg.com/Product/Product.aspx?Item=N82E16813128084
Intel Core 2 Quad Q6600 (OC 3ghz) – http://www.newegg.com/Product/Product.aspx?Item=N82E16819115018
CORSAIR 4GB DDR2-1066 – http://www.newegg.com/Product/Product.aspx?Item=N82E16820145197
Seagate 160GB SATA – http://www.newegg.com/Product/Product.aspx?Item=N82E16822148230
LITE-ON DVD burner – http://www.newegg.com/Product/Product.aspx?Item=N82E16827106263
GeForce 7200GS – http://www.newegg.com/Product/Product.aspx?Item=N82E16814500036
ZALMAN 9500 – http://www.newegg.com/Product/Product.aspx?Item=N82E16835118223

I will overclock the blades to 3ghz, so it will spec at 3ghz quadcore with 4gb of ram, with the option to go 8 gigs. Before anyone says anything, hard drives are not an issue, I will be building a NAS, and neither are video cards. I don’t render with them nor will I have a monitor hooked up to them so really they are just there for just in case scenarios since the mobo does not have video support built in.

Now, here is the challenge…I know I can build 4U servers no problem, (they are pictured in the diagram) but since this rack is a 15U, I want to try and go 3U so I can fit an extra render machine, plus more stuff in there like my web server and a switch and patch panel…Now the real challenge I face is cooling, I need to fuigure out a way how to keep my processors cool at 3U, a regular heatsync like the Zalman 9500 wont fit in that short of a case. Water cooling seems like an impractical and expensive way of doing it, so I may need to think about this a bit longer…I can just put a heat spreader on the processors, and put an extra fan or two in the case (more noise), who knows, let me think about it.

One thing I did think about when writing up this blog is, I should probobly have my server infront of the firewall…I wonder if I should even bother putting it in a rack if thats the case…hmm…Anyway, look forward to another post in the near future…

Keith