Hi guys


As I've said on Bean's NAS topic, I'm going to build one for myself and it has to be upgradeable and it must have the very best networking solution.


I've decided to go with ZFS and RAIDz2 (some sort of RAID6) for multiple reasons, which you can find here: https://calomel.org/zfs_raid_speed_capacity.html


First of all, the most important: networking. Any NAS will be transferring data back and forth through its network interface. Therefore, considering the fact that ZFS provides two levels of high-speed cache (ARC=RAM, L2ARC=SSD), I need a networking solution that is capable of taking advantage of these. A single Gigabit Ethernet link is useless, same goes for two. I could use two 4GbE NICs but it wastes so many ports on the switch... So I'll be going with 10GbE (in the future, too expensive for now and haven't got enough $$ ). The NAS will mainly be serving my own computer, so a direct 10GbE link between the two is perfect (10GbE switches cost a lot). Two or three Gigabit links will serve other computers, in rare occasions. Each 10GbE card runs at PCI-e 8x so let’s consider one port used for that. I could also use double 10GbE NIC for a 20GbE link, but I have time to think about this.





RAM runs at 1333 9-9-9-27 2T (Rampage III Extreme with 990X and ValueRAM DIMMs, shame on me for those timings), and still 10GbE isn’t enough to fully take advantage of the RAM cache (obviously). But anyway, 1GiB/s is still pretty darn fast. The SSD won’t even use that much bandwidth so that’s fine.


Now let’s get down to storage. As I’ll use 2-3 GbE links for now, while also taking into count a dedicated PCI-E 8x port for the future NIC, I still need another 8x port for the disk controller. This one will provide me eight SATA3 ports, so up to 12TB with RAIDz2, with each drive being 2TB. I believe that’s enough space for a quite a few years. When 12TB becomes not enough, I’ll be switching to newer drives anyway.


So in this regard, I only need two PCI-E 8x ports (16 lanes). No need for quad PCI-E 8x capable motherboards; two 8x ports are enough, some 4x are welcome.To have decent network speeds, I’ll be throwing in a PCI GbE NIC to have 384MiB/s until I get the 10GbE stuff.


With this in mind, I chose the following components:


PSU: Seasonic G-450 => Seasonic rocks, solid power units and Gold certification tops it up
Case: BitFenix Shinobi Core => 8 disk bays, two fans for HDD cooling and good cable management. What else ?
CPU: i3 3220 => ECC support (mandatory) and cheapest 2C/4T (good for compression) socket-1155 processor
MOBO: ASUS P8B WS => Good PCI-E layout, PCI slot, ECC support (mandatory)
RAM: ValueRAM 4GB ECC 1333 => cheapest ECC memory module at 1333 or more
SSD: Kingston SSDNow V300 60GB => cheap SSD used only for caching, no need for 2K$ Intel crap
Disk controller: Highpoint RocketRAID 2720SGL => best option after LSI, decent performance and JBOD
Disks: WD RE 2TB => WD hasn’t failed on me yet, and RE drives are solid
Fans: Noiseblocker XL2 Rev.3 => silent fan with decent airflow


I think that's the best way to go. I'll start with 4 drives, so 4TB usable and 2 drives for parity. I'll be using a 1810G-24 switch from HP (24 ports) with dual GbE and one PCI NIC for the server and three out of four ports from my Intel server NIC for my personal computer. For now, that's enough and a bit faster that my normal Caviar Black.

What do you guys think ?

I'll write a real reply late, when I have time. For now... WOW
Niiice

I want to see and read moar!!!

Bean 3930K on RIVE @4.5ghz - 16gb Gskill Ripjaw Z - 780 Ghz Edition - Corsair AX850 - assorted SSD's and Mech drives.

This project sound awesome!

There is just one thought that crossed my mind. If you really want to be able to benefit also from the super high IOPS you get with RAM caching, then you need to make sure, your network equipment can handle latencies as low as 1/3 microseconds. By any chance, do you know the specs for 10 GBit Ethernet regarding latency?

Thanks for the feedback guys

Haven't looked into latency, but right now I've tested pinging my router and it takes 0.5ms (so 500 microseconds).
Found on Wiki about 10GbE: "10GBASE-T has latency in the range 2 to 4 microseconds compared to 1 to 12 microseconds on 1000BASE-T".
Here, it says that 10GbE has got a mean latency of 9usec (http://www.extremenetworks.com/librar ... _Network_Applications.pdf)
Here, it says that GbE (in full-duplex mode) has a latency on 0.5usec with a 64B packet (http://www.datalinkcom.net/How%20Fast ... %20Gigabit%20Ethernet.pdf).
Finally, here it says about 50-125usec for GbE and 5-50 for 10GbE when using TCP/IP (http://www.qlogic.com/Resources/Docum ... n_to_Ethernet_Latency.pdf).

I guess using full-duplex, offloading the load of the NIC (and not the CPU) and using Jumbo-Frames can help a bit with the latency issue. If ever there's any delay, I think the server or the client will adapt itself and wait until the working data set is processed before taking another one.

I could also switch to Fibre Channel but the max speed is 8Gbit per SFP slot. :/ That way I get a little lest bandwidth but light-speed latency.

Yes, usually SFP solves latency related issues. Basically your setup should definitely be sufficient for your needs, except you want to have some network boot solution in place and you want to have performance that is on par with RAM caching in an actual computer

I think the only programs that are time-critical are games, as I'll be loading the from the NAS. Any delay, especially in online games, is intolerable. If there's any, then I'll keep them on my Caviar Black. Else is only movies, music and stuff so that's fine.

Quote:


Wait... Hold up... You're going to be running games from a NAS?
Crazy kat, that's awesome 3930K on RIVE @4.5ghz - 16gb Gskill Ripjaw Z - 780 Ghz Edition - Corsair AX850 - assorted SSD's and Mech drives.

Yeah, RAIDz2 is faster than my Caviar Black I believe. If it is the case, then of course I'll be running games from the NAS (if no delay). :p

Quote:

Running games from the NAS really shouldn't be an issue. The scenario where the networ latency shouldn't be higher than 1/3us is when you're storage system is pumping out 300'000 IOPS! Which btw. is an insane amount of IOPS and a highly unlikely scenario. With standard applications you will never and I mean it, really never need more than 100'000 IOPS. For comparison: you're Caviar Black does 500 IOPS best case.

The only use of such speeds would be faster loading, but yeah no application needs 300K IOPS (maybe something that processes a lot of data in the RAM).