Motherboards with 3 PCIe_x16 slots

[RoDiesel]

Hi to all. Looking to build a NAS, I decided to go for a DIY solution. I like a lot the Asustor 12 Pro (https://www.asustor.com/en/product?p_id=80) that has 12 slots for NVMe storage, but I find the CPU rather slow.
So I need a motherboard that supports 3 PCIe_x16 (x4 x4 x4 x4 for bifurcation) to install 3 pieces of quad nvme adapter (eg. https://vi.aliexpress.com/item/1005005573990685.html?gatewayAdapt=glo2vnm) for a total of 12 drives. Also, 10 Gb Ethernet would be fine. I can also go with just 2 PCIe_x16, for a total of 8 drives.
For the CPU, I have in mind the 5600G or 5700G. But I'm open to any decent CPU. Integrated graphic would be a plus.
As an alternative, I could go the SSD way, but considering the price difference / TB, the NVMe makes more sense.
Any idea? The budget for the build must be something in the value of Asustor 12 Pro.
Regards

 

[kapone32]

There are a few issues with your request.

1. There are no boards that support what you want on AM4. Even the MSI X570S Ace Max is the best wired board for AM4 but the PCIe allocation is 16/0/4 or 8/8/4.
2. The 5600G and 5700G both only support 8 lanes on the first slot, even if the first slot support x16.
3. AM5 may be able to help as you get 2 GPU cores on all their chips. So the 7600 would be the cheapest way to get into it. The board I am thinking is the MSI X670E Carbon as that is one of the only boards that supports NVME on all the PCIe lanes 8/8/4. Another board would be the Asus X670E E Strix as that has support for x8 on the first slot. Use a x4 in the 2nd slot and a card like the WD AN1500 in the 3rd slot. You get 4 M2 slots as well.
4. Going past 2 drives brings diminshing returns in real world use.


All of this is moot if you have the money to get into HEDT as Threadripper is currently the easiset way to get what you want.

I can understand your want as well. I have 25 TB of NAND Flash in my PC and it sings.

If you don't vare about ultimate performance X399 can be decent if you are willing to buy used MBs. I do believe Amazon still sells the 1900X for like $240. With that you get 64 lanes and boards like the As Rock X399 Challenger give you 3 x16 slots do to with as you please.

 

[Fouquin]

2. The 5600G and 5700G both only support 8 lanes on the first slot, even if the first slot support x16.

Not according to AMD. But then they never bothered to adjust their official spec pages on any of these chips and just leave it up to consumers to buy it and figure things out for themselves.

 

[kapone32]

Not according to AMD. But then they never bothered to adjust their official spec pages on any of these chips and just leave it up to consumers to buy it and figure things out for themselves.

I have a 5600G and can confirm that you get 8 lanes if you use the IGPU. It may do 16 when doing a DGPU but MBs do state that on every AM4 board that 8 lanes are dedicated to the IGPU if you are using it to be the main display. You can even dedicate up to 16GB of the RAM strictly for the IGPU on 5000 APUs. The 3400G did not support this.

 

[damric]

4-Drive M.2 NVMe SSD to PCIe 3.0 x4 Adapter Card

⦁ Add up to four NVMe SSDs using only one PCIe slot.⦁ Designed for use with M.2 2280 form factor SSDs.⦁ Compatible with PCIe 3.0 motherboards and backward compatible with PCIe 2.0 / 1.0.⦁ Aluminum construction for maximum heat dissipation and optimal performance.⦁ Ample support for legacy...

sabrent.com

This is the more expensive adaptor that doesn't need the lane splitting. A few of these might solve your problem.

 

[RoDiesel]

What do you think about this MB: https://www.gigabyte.com/Motherboard/X399-AORUS-XTREME-rev-10#kf combined with AMD Ryzen Threadripper 1900X
By the way, very informative details about the 5600G

4-Drive M.2 NVMe SSD to PCIe 3.0 x4 Adapter Card

⦁ Add up to four NVMe SSDs using only one PCIe slot.⦁ Designed for use with M.2 2280 form factor SSDs.⦁ Compatible with PCIe 3.0 motherboards and backward compatible with PCIe 2.0 / 1.0.⦁ Aluminum construction for maximum heat dissipation and optimal performance.⦁ Ample support for legacy...

sabrent.com

This is the more expensive adaptor that doesn't need the lane splitting. A few of these might solve your problem.

Interesting idea. But quite expensive ...

 

[damric]

What do you think about this MB: https://www.gigabyte.com/Motherboard/X399-AORUS-XTREME-rev-10#kf combined with AMD Ryzen Threadripper 1900X
By the way, very informative details about the 5600G


Interesting idea. But quite expensive ...

I understand that, but ultimately there's no better choice because this unties your hands for the rest of your platform. You can do it one piece at a time as budget allows.

 

[InVasMani]

If it were me in that situation I'd probably get a z790 Nova and something like a 12700K or 12600K on sale to pair with it. DDR5 for a modest kit is inexpensive these days. It's got 6 M.2 slots on the board itself along with a 5.0 x16 and 4.0 x16 slot and sure the top M.2 is Gen 5, but backwards compatible with Gen 4. So that's like 10 to 14 M.2's if you don't intend to use a discrete GPU. It even has the external clock generator to OC alder lake chip parts. Has a thunder bolt header along with wifi 7 and 2.5Gbe. It's even got a few SATA ports so drop the OS on those and use the M.2s in a raid if you wish.

 

[unwind-protect]

What do you think about this MB: https://www.gigabyte.com/Motherboard/X399-AORUS-XTREME-rev-10#kf combined with AMD Ryzen Threadripper 1900X
By the way, very informative details about the 5600G


Interesting idea. But quite expensive ...

Yeah, if you really need 12 NVMe then going with a server or HEDT platform cuts through the crap quite nicely.

Even Xeon V4s make good modern fileservers.

 

[kapone32]

What do you think about this MB: https://www.gigabyte.com/Motherboard/X399-AORUS-XTREME-rev-10#kf combined with AMD Ryzen Threadripper 1900X
By the way, very informative details about the 5600G


Interesting idea. But quite expensive ...

Gigabyte were the worst X399 boards. The Asus Strix is better but a bit limited in support.

If it were me in that situation I'd probably get a z790 Nova and something like a 12700K or 12600K on sale to pair with it. DDR5 for a modest kit is inexpensive these days. It's got 6 M.2 slots on the board itself along with a 5.0 x16 and 4.0 x16 slot and sure the top M.2 is Gen 5, but backwards compatible with Gen 4. So that's like 10 to 14 M.2's if you don't intend to use a discrete GPU. It even has the external clock generator to OC alder lake chip parts. Has a thunder bolt header along with wifi 7 and 2.5Gbe. It's even got a few SATA ports so drop the OS on those and use the M.2s in a raid if you wish.

That seems to be more lanes than the CPU provides.

 

[TheoneandonlyMrK]

Not according to AMD. But then they never bothered to adjust their official spec pages on any of these chips and just leave it up to consumers to buy it and figure things out for themselves.

They do have a x16 main GPU slot, one attached to CPU nvme and four lanes to a South bridge.

But they bifurcate poorly, no Am4 chip or board will split a x16 into 4x4 nvme (in a useful way I've no GPU and still only one x16 slot)I tried with a Asus addin card with four in it, the only way to do it is only on the main slot and with a igpu.

If the board has a second x16 any use of it drops it's slot and pciec1 slot to x8 x8.

You would need thread ripper, for what the OP suggests.

Oh and Consumer Intel does NO better at either bifurcation or lane allowance.

Server really is the only way.

@kapone32
Igpu use no external pciecx lanes in use or not, I have a 5600G right here.

 

[InVasMani]

Lane support slipped my mind entirely you're right Alder Lake is 20 PCIE CPU lanes. So the lane usage works a bit different from what I was thinking. I don't know why I was thinking the M.2's on board didn't chew into PCIE lanes, but surprise they do it appears. Pretty messy with all the PCIE gen change transitioning too up to a point.

 

[RoDiesel]

They do have a x16 main GPU slot, one attached to CPU nvme and four lanes to a South bridge.

But they bifurcate poorly, no Am4 chip or board will split a x16 into 4x4 nvme (in a useful way I've no GPU and still only one x16 slot)I tried with a Asus addin card with four in it, the only way to do it is only on the main slot and with a igpu.

If the board has a second x16 any use of it drops it's slot and pciec1 slot to x8 x8.

You would need thread ripper, for what the OP suggests.

Oh and Consumer Intel does NO better at either bifurcation or lane allowance.

Server really is the only way.

@kapone32
Igpu use no external pciecx lanes in use or not, I have a 5600G right here.

Good points and well spotted.
Maybe we should just drop some of the requirements. Let's go for 8 NVMe drives for storage + 1 for OS (6 + 1 is the worst case scenario). No GPU needed, as is web managed. And with the use of a decent MB with a CPU that is not 180 TDW (as is the 1900x). Preferably around 65w TDP. Cooling and especially the noise might be a concern. The example with the 5600G was also some kind of acceptable levels of power consumption and computing power. Z790 maybe?

 

[InVasMani]

Seems like you just need Z690 with four M.2 slots mostly with either Alder Lake or Raptor Lake CPU of whatever SKU fits you're expectations. Just power limit it and undervolt a little if worried about the power. Besides these aren't the higher end unlocked i7/i9 SKU's. You might do alright with like 13400 or 13500 and z790 board. That would allow for more Gen 4 M.2's to raid if that's what you had in mind.

 

[evernessince]

Good points and well spotted.
Maybe we should just drop some of the requirements. Let's go for 8 NVMe drives for storage + 1 for OS (6 + 1 is the worst case scenario). No GPU needed, as is web managed. And with the use of a decent MB with a CPU that is not 180 TDW (as is the 1900x). Preferably around 65w TDP. Cooling and especially the noise might be a concern. The example with the 5600G was also some kind of acceptable levels of power consumption and computing power. Z790 maybe?

How much SSD storage do you need total and what's the use case? Quad M.2 SSD cards typically run $75. If you need bifurcation, you either have to spend $160 per card or upgrade to an HEDT platform (which is super overkill for a NAS as I'll explain later). Otherwise you can just run all four SSDs per adapter as a JBOD array, which isn't a big deal if all you need is one big storage pool. Also the more expensive NVMe PCIe adapters don't need x16 slots, there are x4 models that will simply share the x4 among all the SSDs. That usually isn't a problem unless you have a very demanding use case in mind.

Of course if all you need is a ton of fast SSD storage you can purchase M.2 to U.2 adapters and get enterprise drives very cheaply. 15.36TB drives go for $1,050 regularly on eBay and their endurance will be more than enough. 8TB enterprise drives go for around $300 - 500. That's way better than any consumer options, even the 870 QVO is current sitting at $500 and that'll have a fraction of the endurance.

A single PCIe Gen 3 drive will be able to more than saturate 10 Gb ethernet so there'd really be no advantage to having a HEDT platform with a ton of PCIe lanes. You'd be limited to 1.25 GB/s (and that's ignoring trasmission overhead) with a 10 Gbps nic. If you are only using those lanes to connect more storage, it's far better to just buy a larger enterprise drive, which go all the way up to 30.72 TB right now.

 

[kapone32]

How much SSD storage do you need total and what's the use case? Quad M.2 SSD cards typically run $75. If you need bifurcation, you either have to spend $160 per card or upgrade to an HEDT platform (which is super overkill for a NAS as I'll explain later). Otherwise you can just run all four SSDs per adapter as a JBOD array, which isn't a big deal if all you need is one big storage pool. Also the more expensive NVMe PCIe adapters don't need x16 slots, there are x4 models that will simply share the x4 among all the SSDs. That usually isn't a problem unless you have a very demanding use case in mind.

Of course if all you need is a ton of fast SSD storage you can purchase M.2 to U.2 adapters and get enterprise drives very cheaply. 15.36TB drives go for $1,050 regularly on eBay and their endurance will be more than enough. 8TB enterprise drives go for around $300 - 500. That's way better than any consumer options, even the 870 QVO is current sitting at $500 and that'll have a fraction of the endurance.

A single PCIe Gen 3 drive will be able to more than saturate 10 Gb ethernet so there'd really be no advantage to having a HEDT platform with a ton of PCIe lanes. You'd be limited to 1.25 GB/s (and that's ignoring trasmission overhead) with a 10 Gbps nic. If you are only using those lanes to connect more storage, it's far better to just buy a larger enterprise drive, which go all the way up to 30.72 TB right now.

I have not though about that route but X399 is not expensive right now. You have to agree that a $200 CPU and $300 used board would rival that. Enterprise drives on Ebay are used and one of the tenets of DIY is never buy storage on Ebay lol. Not saying what you have does not have merit but they are still used.

 

[unwind-protect]

I have not though about that route but X399 is not expensive right now. You have to agree that a $200 CPU and $300 used board would rival that. Enterprise drives on Ebay are used and one of the tenets of DIY is never buy storage on Ebay lol. Not saying what you have does not have merit but they are still used.

8 TB new:

Intel DC P4510 SSDPE2KX080T851 0M6CNF 8TB PCIe Gen 3.1 X4 4GB/s 3D TLC 1DWPD U.2 NVMe 2.5in Solid State Drive

Buy the Intel DC P4510 8TB PCIe Gen 3.1 X4 2.5" SSD SSDPE2KX080T851 at the lowest price with free & fast shipping. Get volume discounts and excellent customer service. Shop now!

serverpartdeals.com

 

[evernessince]

I have not though about that route but X399 is not expensive right now. You have to agree that a $200 CPU and $300 used board would rival that. Enterprise drives on Ebay are used and one of the tenets of DIY is never buy storage on Ebay lol. Not saying what you have does not have merit but they are still used.

You can buy new or used. Serverpartsdeals offer the same products on their own store as they do on eBay. Mind you I've never had a problem with any of the storage devices I've purchased off eBay. If you are buying used you just need to make sure you are getting the SMART info before purchasing and testing the drive when it comes in but you should be doing the latter regardless given the number of fakes on Amazon nowadays. It's very hard to get a bad product when you do this, especially for an SSD that isn't as fragile as a HDD and is not so easily damaged during shipping. Used SSDs in the enterprise space aren't something to avoid. In many cases less than 10% of an enterprise drive's life is used up by the time it's decommissioned. In the case of the 9300 Pro for example, that still leaves you with 60 PB of writes, which is vastly more than you'll get with any new consumer drive.

Depending on OP's use case, it may make more sense to just slap an enterprise drive in there. I don't think we should just disqualify certain options because of an eBay bad mentality.

 

[freeagent]

I have one of those Asus Hyper M.2 Gen 4 cards, and with B550 or X570 if I have GPU slotted, I can only run 2x NVME. Top slot, bottom slot.. doesn't matter.

 

[TheoneandonlyMrK]

I have one of those Asus Hyper M.2 Gen 4 cards, and with B550 or X570 if I have GPU slotted, I can only run 2x NVME. Top slot, bottom slot.. doesn't matter.

Same only x16 was possible on a couple of board's usually by bottom slotting a GPU via the chip set, to allow the x16 to go x8x8 in slot 1

And those boards usually had two pciecx x16 that do x8 x8 when both in use so Meh , I I obviously raid 0d 4 just to see.once in a Asus hyperx16 , it wasn't practical in Am4 with a GPU though.

 

[evernessince]

I have one of those Asus Hyper M.2 Gen 4 cards, and with B550 or X570 if I have GPU slotted, I can only run 2x NVME. Top slot, bottom slot.. doesn't matter.

Yep, cards like the ASUS are dumb cards in the sense that they are simply adding M.2 slots without adding a controller that would allow for bifurcation, lane sharing, ect. M.2 already directly interfaces via PCIe as well so it's not like they are doing anything special in that regard, $75 for that is grossly overpriced IMO. Unfortunately cards with a proper controller start at $150 (for 4 slot PCIe x4 although all four drives can operate at up to x4) unless you buy used. Neither option is really worth it given the price unless you alreaady have a motherboard that can take advantage of it. Just buying a larger enterprise drive is usually cheaper, unless you need something like 30 TB plus drives, those have a large premium.

 

[kapone32]

Yep, cards like the ASUS are dumb cards in the sense that they are simply adding M.2 slots without adding a controller that would allow for bifurcation, lane sharing, ect. M.2 already directly interfaces via PCIe as well so it's not like they are doing anything special in that regard as well, $75 for that is grossly overpriced IMO. Unfortunately cards with a proper controller start at $150 (for 4 slot PCIe x4 although all four drives can operate at up to x4) unless you buy used. Neither option is really worth it given the price unless you alreaady have a motherboard that can take advantage of it. Just buying a larger enterprise drive is usually cheaper, unless you need something like 30 TB plus drives, those have a large premium.

Exactly the first part. I remember buying my first Hyper X card for $45 Canadian. They made complete sense for Threadripper for the average consumer. At the time 1 TB drives vs 2 TB were 3 x more expensive. In a lot of cases you could build a 4 TB array using 4 drives. I had 4 Intel 660Ps in one and saw almost 7000MB/s writes using X399. I am really liking what you are saying about Enterprise drives though. I think my board might even have a U2 port.

 

[evernessince]

Exactly the first part. I remember buying my first Hyper X card for $45 Canadian. They made complete sense for Threadripper for the average consumer. At the time 1 TB drives vs 2 TB were 3 x more expensive. In a lot of cases you could build a 4 TB array using 4 drives. I had 4 Intel 660Ps in one and saw almost 7000MB/s writes using X399. I am really liking what you are saying about Enterprise drives though. I think my board might even have a U2 port.

It's definitely possible, I know some of the HEDT / Professional boards get 1-2 U.2. I really wish some consumer board did as well given that SATA is pretty much being phased out and you have to pay a premium for high density M.2 given the small form factor. U.2 is that balance of form factor, price, and performance.

 

[RoDiesel]

A lot of options but still in doubt about the HW purchase (new or used). The NAS is needed for a small office (~10 users, for the moment). The plan for NAS is like this :
- volume 1 for pictures/short movies (at least 2x8 TB),
- volume 2 for documents (at least 2x4 TB) and
- volume 3 for different mixed files (2x4 TB).
So I have 3 volumes, all with Raid1 (6 drives in total).
In case of a HW failure (NAS or drive), I need to be able to extract any healthy drive and read it into some other device/computer (hence the need for mirroring).
For volume 2, I have to store/read a lot of small files and also be able to search inside them. This is one of the reason I go for a DIY NAS, with a much powerful CPU than the N5105.
On the NAS I will also install some add-on packages/virtualization.
I could go with 2 smaller NVMe drives at the beginning (2 x 8 TB) for all 3 volumes, but in time, the full version will still need to have at least 6 NVMe drives.
I was able to find a used x399 MB with a 1900x on it, at a decent price. But the TDP of the CPU is a no go at this point.
Any thoughts?

 

[Frick]

What are the use cases, specifically? Will the files be worked on from the NAS by multiple people, or is it pure storage?