NVIDIA GeForce RTX 4090 PCI-Express Scaling

[Mussels]

Uh I didn't say that? 6?? Not sure what you read here. I said it's either going to run at PCIE4x2 or PCIE4x4, the manual makes it look like the slot will run at PCIE4x2 no matter what, even if the NVME slot is empty, which is dumb, but unsure if a BIOS update in the future could just see the slot run 4x4 when no NVME installed - depends on how it's wired? And the manual doesn't tell me that or if it does I didn't see it.

You're missing the point that these are physically wired. You cant use the BIOS to change that.
You want more slots and less NVME - go buy a board that has that internal wiring set up to do so.

 

[pavle]

Depending on amount of data that's being transferred via PCI-Express, but if you got 2.0 x16 you're good to go, one could easily live with 1.1 though.

 

[JimGrongle]

You're missing the point that these are physically wired. You cant use the BIOS to change that.
You want more slots and less NVME - go buy a board that has that internal wiring set up to do so.

Ah cool well that's what I didn't know. I couldn't tell if it was physically limited to only ever being at 4x2. I thought if the PCIE and M2 were connected they'd be sharing lanes, but my assumption is wrong I see. Thanks for pointing that out!
The additional of a billion NVMEs does get me down. I'm coming from a board with 7 slots (on broadwell) 4 of which will run at PCIE3x16. And now the most a board has is 3, and the third is often crippled. It's terrible!

Depending on amount of data that's being transferred via PCI-Express, but if you got 2.0 x16 you're good to go, one could easily live with 1.1 though.

For some things indeed, for games or heaven forbid 3D workflows, not so much.

 

[Mussels]

Ah cool well that's what I didn't know. I couldn't tell if it was physically limited to only ever being at 4x2. I thought if the PCIE and M2 were connected they'd be sharing lanes, but my assumption is wrong I see. Thanks for pointing that out!
The additional of a billion NVMEs does get me down. I'm coming from a board with 7 slots (on broadwell) 4 of which will run at PCIE3x16. And now the most a board has is 3, and the third is often crippled. It's terrible!


For some things indeed, for games or heaven forbid 3D workflows, not so much.

Intel shared lanes from the chipset because they had such a shortage of them it's all they could do
It wasn't ever done to be user friendly or helpful, but as a way to get the features users wanted and hoping they didn't ever use them all at once.

AM5 vs intel at present, AMD simply has 4x more lanes on the CPU, 28 5.0 on AM5 vs 16 5.0 + 4 3.0 on 13th gen intel.
Where those x4 lanes go is upto the board maker, but they cant be mixed with lanes from the chipset

I'm having trouble finding the exact wording to explain why this is, it seems obvious to me but i also can't word it well - it's like getting a USB hub and plugging it back into itself, hoping to get more wattage and speed.

 

[JimGrongle]

All good, I am understanding this more, as I read more and more board reviews and GPU reviews.
Probably a lot of people's confusion comes from the update cycle - for myself I was on X99 with 7 slots and 48 lanes (40? I forget) and everything was great. Boards cost half as much as they do now, and still came with dual 10gbe and 4 of the 7/8 PCIE slots could all run at 3x16. Now we get two slots, 5 NVMEs and 2.5gb LAN for 2x the price. My confusion and frustration feed off each other. And around we go.
But assuming others are like me and coming from the 10xx or 20xx generation of GPUs (and matching boards), well things have obviously changed a lot and not so much for the better, for those of us looking to use these machines for work and not games. In 3D where CUDA cores are king, Quadros just don't make any sense any more unless you work in CAD or maybe Architecture visualisation. For anyone creating 3D content, all we want is the most GPUs in a single box for the lowest price.

This article has been an absolute lifesaver though (now that I have a better understanding of the current state of affairs) in learning how I can evaluate different systems and generations, even down to how my X99 board will handle some 4090s (and I plan to throw some on it!) compared to a new rig.

 

[Mussels]

Lanes back then were all from the chipset, back before everything was integrated into CPUs
It wasnt as fast as it seemed because you might have had four 16x PCI-E slots but up-stream they'd all have a singular link back to the CPU choking them all anyway


x58 had "36 Lanes, 4x8, and 1x4" from the chipset with 25.6GB/s bandwidth on the QPI link from chipset to CPU
12.8GB/s each direction on the QPI, while the PCI-E lanes totaled 18GB/s - there was never enough bandwidth for everything at the same time

Eventually i guess they made the decision to just cut devices off entirely, with a method to choose one or the other instead of competing devices (The good old days of your network card causing your audio to crackle, etc)

 

[JimGrongle]

Ah well that explains some of this, I didn't realise they were all from the chipset on those older generations! Thanks for the explanation, the detail you've provided me in these replies has really shown me a lot, and it's much appreciated.

I will have to do a bandwidth test of some kind on my older x99 board to see how they're running. Going to replace 4x1080s with 2x4090s on that board, parts are ordered (and pushing a new system build to the new year).

On the plus side, going from 4 GPUs down to 2, sounds like it will be better for any potential bottlenecks I may have had with 4 GPUs fighting for data. Will run some tests to find out, before and after upgrade to compare.

 

[Mussels]

To be fair i had to google it to get accurate information myself, i just had a heads up remembering some of the keywords and acronyms

You are aware there is no such thing as SLI any more?

 

[JimGrongle]

To be fair i had to google it to get accurate information myself, i just had a heads up remembering some of the keywords and acronyms

You are aware there is no such thing as SLI any more?

Yes, for 3D rendering I don't need SLI, it's really down to how many CUDA cores there are. More cores, faster render. I mean SLI would be nice for machine learning maybe, and that's becoming a bigger thing, but my own day to day work is not in that field - I just need renders out the fastest possible, and right now that means 4090s, whatever I can get hold of!

 

[Tek-Check]

if the AMD board would see a 3rd slot speed of 4x2 with zero NVMEs installed,

Dude, open .pdf manuals of both boards on Asus website, and under each board you will find literally all answers to your questions, with simple diagrams showing which slots share lanes and which do not.

AM5 vs intel at present, AMD simply has 4x more lanes on the CPU, 28 5.0 on AM5 vs 16 5.0 + 4 3.0 on 13th gen intel.
Where those x4 lanes go is upto the board maker, but they cant be mixed with lanes from the chipset

Check the chipset link speeds in diagrams in CPU data base.
AM5 - 24+4 Gen5, x4 lanes reserved for the chipset link
Z790 - 16 Gen5 + x4 Gen4 and x8 Gen4 for DMI chipset link.

Both CPUs have the same amount of PCIe lanes (28), but differently distributed and of different generation.
Although AMD's AM5 platform has more usable Gen5 lanes, Intel's chipset link to CPU is twice as fast comparing to AM5 (15.7 GB/s vs 7.6 GB/s).
This means that if you copy data from two NVMe drives on Intel's chipset somewhere to CPU, you will not have speed bottleneck. With AM5, you will, so it's important to be aware of bottlenecks.

 

[Mussels]

This means that if you copy data from two NVMe drives on Intel's chipset somewhere to CPU, you will not have speed bottleneck. With AM5, you will, so it's important to be aware of bottlenecks.

Yes, chipset connected ones would be slower - but AMD have two 5.0 NVME slots direct from the CPU to avoid that

What does stand out is that AM4 had SATA on the CPU while AM5 is PCI-E only, so SATA devices could be slower on AM5 while NVME shouldn't

Board makers could slap that 4x somewhere else if wanted, but i'd be surprised if they did
The next gen chipsets using a 5.0 link would catch up on chipset connected devices

 

[Tek-Check]

AMD have two 5.0 NVME slots direct from the CPU to avoid that

True that, but only if two drives communicate with each other. Besides, I have seen just a few double NVMe Gen5 drive implementations.

next gen chipsets using a 5.0 link would catch up on chipset connected devices

Hopefully. Promontory 21 chipset does need to evolve for this to happen.