HELP! Quad RTX 3090, x2 EVGA 1600 G2: Burnt out +12v on MoBo power connector

[FrankJames]

Hi Guys,

New to this forum.

I just built a custom rig for Deep Learning and after 2-3 months of constant use, I noticed the 2 +12v lines from PSU to MoBo have burned up. Just took it all apart, MoBo is fine, cable is not, +12v pins burnt in place on PSU 1.

SPEC:
- ROG zenith extreme alpha ii
- Threadripper 3990x
- Trident Z royal 32 gig x8
- 4x ASUS RTX 3090 OC (using PCIe X16 riser cables)
- EVGA 1600 G2 x2
- x3 EK waterblock/pumps
- GPUs & CPU are water cooled
- Im using 3 seperate cases, one big center one to house all the main stuff, 2 smaller side towers (all mounted on a wall) to house cooling towers & PSUs

POWER SET UP:
- MoBo, CPU, GPU 1&2 + all fans & pumps on PSU 1.
- GPU 3&4 on PSU 2
- Both PSUs are on one dedicated back up battery to each PSU plugged into the same wall circuit. The battery back-ups are at ~60% max load when GPUs are running flat out.
- A little light overclocking on the RAM in BIOS, nothing serious.
- Temps: Cards run around 50-60 degrees during processing, CPU runs around 55-65 at full power.

CONSUMPTION
- Scenario 1: Im typicially running Neural Net training for 4-8 hours at a time, pulling around 275 W from each card. This is not CPU intensive at all.
- Scenario 2: Occasionally we will run hypotheses simulations through the Threadripper, getting it up to 90% of power usage ~400 W but not at the same time as scenario 1

Thoughts so far:
From everything I have read, it looks like i'm overloading the PCIe power bus on the MoBo.
- I can't add a PCIe power booster as I have all 4 slots in use.
- Potential options: Undervolt cards, pull-out 10-20% performance
- Can you reduce or cap-out the PCIe power draw or cut it completely from the BIOS? If the 2x 8 Pin power connectors on GPUs provide 150W each, should i maybe limit each card to say 280W to stop them pulling from PCIe power bus?


HELP!!! Need to get back to work!

 

[eidairaman1]

Your current draw was too much

 

[newtekie1]

Did you plug in the 4-pin molex connector at the bottom of the motherboard?

 

[pantherx12]

Can you tell us precisely what you plugged into motherboard?
Or better yet a picture of everything connected

 

[Chomiq]

Those aren't the stock cables that come with that PSU:

 

[Ominence]

for such an expensive system, i would suggest using stock supplied power cables. your sleeved cables may look good but might be 'cheap' on the inside 'core' meaning AWG.
please ensure that you are not overloading too much current draw on your cooling setup directly from the MB 4 pin PWM connectors including high draw pumps.
you are very poorly load balanced. 3x of the 3090s should be on PSU2- not just 2.
please ensure all 5 power connectors to the MB are fully connected. no 4 pin CPU shortcuts for example.
please do not overclock the CPU and RAM excessively. limit the voltage and current draw equation you're running.
please ensure that your power cable runs are not too long. the longer they are, the more resistive heat is likely to build at the connector interface.
BIOS does not appear to support configurable PCIe power limits.

 

[Chomiq]

12:40

 

[Ominence]

- Potential options: Undervolt cards, pull-out 10-20% performance
- Can you reduce or cap-out the PCIe power draw or cut it completely from the BIOS? If the 2x 8 Pin power connectors on GPUs provide 150W each, should i maybe limit each card to say 280W to stop them pulling from PCIe power bus?

325w x4 should be an absolute non issue just on the GPUs. the MB should be able to handle combined 100w on the PCIe quite easily. nvidia load balances how much draw can occur over PCIe and the algorithm is quite conservative.

 

[lowrider_05]

Yeah, as some mentioned before, those sleeved cables are not designed to handle long sustained loads. Use the stock cables and you should be fine.

 

[kayjay010101]

Have to agree with @Chomiq, looks like cheap sleeved cables that weren't able to handle the current draw.

 

[FrankJames]

Did you plug in the 4-pin molex connector at the bottom of the motherboard?

Yep, sure did!

Can you tell us precisely what you plugged into motherboard?
Or better yet a picture of everything connected

Hey! Yes I have pics during build and after complete. LMK if you can see what you need

Those aren't the stock cables that come with that PSU:
View attachment 208866

Good point, will try this as part of the fix

Can you tell us precisely what you plugged into motherboard?
Or better yet a picture of everything connected

wait, took this video, this should cover everything (screenshots)

for such an expensive system, i would suggest using stock supplied power cables. your sleeved cables may look good but might be 'cheap' on the inside 'core' meaning AWG.
please ensure that you are not overloading too much current draw on your cooling setup directly from the MB 4 pin PWM connectors including high draw pumps.
you are very poorly load balanced. 3x of the 3090s should be on PSU2- not just 2.
please ensure all 5 power connectors to the MB are fully connected. no 4 pin CPU shortcuts for example.
please do not overclock the CPU and RAM excessively. limit the voltage and current draw equation you're running.
please ensure that your power cable runs are not too long. the longer they are, the more resistive heat is likely to build at the connector interface.
BIOS does not appear to support configurable PCIe power limits.

- Pump Draw: So the Pumps (while 12v) are 23W max load. We never max them out at full load so I would say around ~60W maximum. But I have an active ROG pump splitter in the back (seen in image 1830 at the bottom left) so I doubt its that?
- Fan Draw: Same; has an active splitter powering all fans.
- Load balancing: Interesting. So I have calculated that with all pumps, fans, MoBo, CPU pull around 300-400w when active. So with current set up im estimating load at max GPU usage to be:
PSU 1: 1100-1300 W
PSU 2: 700 W
If I load up 3 GPUs on PSU 2 I will be looking at:
PSU 1: 800-1100W
PSU 2: 750 W
Feel like its negligible difference?
- Overclocking: CPU has no OC, RAM is minor but will back
- Power: I'm pretty sure RAM and Threadripper does not use the 12v rail. From what I understand, the GPUs, Fans & Pumps all use the 12v rail but nothing else. Since only the 12v pins on the main MoBo cable were burned out, and all the fans and pumps are on Molex power from PSU 1 (with no issue) it leads me to think its a PCIe power bus overload (runs on 12v). I do have 2 SSds, one in DIMM 2 one in an M2.2 slot and im pretty sure DIMM 2 is pulling from the PCIe bus bout doubt its significant.

325w x4 should be an absolute non issue just on the GPUs. the MB should be able to handle combined 100w on the PCIe quite easily. nvidia load balances how much draw can occur over PCIe and the algorithm is quite conservative.

Yes but a combined PCIe bus draw is well above 100w. Did some more research and the 3090 OC can pull up to 400w at max-out. if the 2 8pin per GPU provide 300W combined power and If thats the case (even during transients) then im looking at a combined 400w pull from the PCIe bus JUST from the GPUs... Am i wrong?

Yeah, as some mentioned before, those sleeved cables are not designed to handle long sustained loads. Use the stock cables and you should be fine.

That makes sense, will swap to all the stock cables. I'm pretty sure the AWG is a much higher rating on the stock. My only concern is regarless of this, I'm not sure I like the fact that i'm putting so much strain on the 12v rail regardless. The PSU can handle it fine but (I Think) but im concerned the MoBo & its PCIe bus is under serious strain.I Know the Zenith extreme Alpha is a beast but i'm looking for long-term reliability.

So:
- Swap out all the shitty power cables with stock
- Undervolt the cards?
- Perhaps I shift the active fan & pump splitters to PSU2 just to ensure a load reduction on PSU1 12v? If I think about it:
- PSU1 12v rail running: GPU 1 & 2, MoBo 12v (PCIe bus), active fan splitter, active pump splitter.
- PSU2 running: GPU 3 & 4.
- Only problem is im a litle concerned about having the MoBo (fan & pump headers) on one PSU and the splitters on another.....

Your current draw was too much

Hey thanks for this! Do you mean on the PCIe power bus?

 

[Ominence]

- Pump Draw: So the Pumps (while 12v) are 23W max load. We never max them out at full load so I would say around ~60W maximum. But I have an active ROG pump splitter in the back (seen in image 1830 at the bottom left) so I doubt its that?
- Fan Draw: Same; has an active splitter powering all fans.

if pumps and fans are not directly running off the MB, then that aspect can be discounted as the one causing the issue.

- Load balancing: Interesting. So I have calculated that with all pumps, fans, MoBo, CPU pull around 300-400w when active. So with current set up im estimating load at max GPU usage to be:
PSU 1: 1100-1300 W
PSU 2: 700 W
If I load up 3 GPUs on PSU 2 I will be looking at:
PSU 1: 800-1100W
PSU 2: 750 W
Feel like its negligible difference?

i would still suggest getting 6 of the 8-pin pcie power connectors on PSU2 to get the max 900w off PSU2. as you mentioned next that the 3090 can pull a transient load of 400w. your research would also tell you that those 8pin pcie connectors do well over 150w at times.

- Overclocking: CPU has no OC, RAM is minor but will back

running RAM at XMP equivalent should really not be an issue.

Yes but a combined PCIe bus draw is well above 100w. Did some more research and the 3090 OC can pull up to 400w at max-out. if the 2 8pin per GPU provide 300W combined power and If thats the case (even during transients) then im looking at a combined 400w pull from the PCIe bus JUST from the GPUs... Am i wrong?

so checking again the 3090 is rated for 350w stock. the 8-pin pcie spec is maxed out at 300w even though i'm sure that the card can draw more out from them as required. they rarely overstress the pcie slot power when are already externally powered. that being said, it might be prudent in your use case to back off the cards to nvidia spec base and boost speeds so that you're in theory then only drawing up to 50w max per slot i/o near 75w. then you'll have a lower draw at the MB cable PSU side of 200w i/o up to 400w.

then you need to know that heat built up over time will expand your cable side connection female end, and lead to potential slowly growing problem of heat and arcing. so your cable side connectors can always be an issue, if not correctly rated.

next, huge conjecture on my part but maybe the PSU isn't the best suited for your 24x7 operation. if the distribution back plane on the PSU burned out due to it being somewhat overstressed and not cutting off in the process, that's a bit of a worry. having said that, you definitely need to get a better and correctly rated 24pin MB cable so that perhaps the poor connection issue at the PSU connector side can be avoided in the 1st place.

your setup is quite elaborate, please ensure that you have sufficient physical strain relief on the PSU side MB cable so that it's not tugging or potentially coming loose over time.

don't bunch up the cables too tightly as that leads to further trapped heat that will increase the overall running temp of the power cables. in your 24x7 environment, that heat may potentially have no 'room to breadth' and cool off or have the right thermal management scope.

 

[thesmokingman]

3090s can draw well over 350w though. Those specific cards the OP is using can draw 420w peak gaming and when really pushed 450w ish. And gaming loads are not constant so if you're slamming them with a constant workload... yikes.

 

[FrankJames]

if pumps and fans are not directly running off the MB, then that aspect can be discounted as the one causing the issue.

i would still suggest getting 6 of the 8-pin pcie power connectors on PSU2 to get the max 900w off PSU2. as you mentioned next that the 3090 can pull a transient load of 400w. your research would also tell you that those 8pin pcie connectors do well over 150w at times.

running RAM at XMP equivalent should really not be an issue.

so checking again the 3090 is rated for 350w stock. the 8-pin pcie spec is maxed out at 300w even though i'm sure that the card can draw more out from them as required. they rarely overstress the pcie slot power when are already externally powered. that being said, it might be prudent in your use case to back off the cards to nvidia spec base and boost speeds so that you're in theory then only drawing up to 50w max per slot i/o near 75w. then you'll have a lower draw at the MB cable PSU side of 200w i/o up to 400w.

then you need to know that heat built up over time will expand your cable side connection female end, and lead to potential slowly growing problem of heat and arcing. so your cable side connectors can always be an issue, if not correctly rated.

next, huge conjecture on my part but maybe the PSU isn't the best suited for your 24x7 operation. if the distribution back plane on the PSU burned out due to it being somewhat overstressed and not cutting off in the process, that's a bit of a worry. having said that, you definitely need to get a better and correctly rated 24pin MB cable so that perhaps the poor connection issue at the PSU connector side can be avoided in the 1st place.

your setup is quite elaborate, please ensure that you have sufficient physical strain relief on the PSU side MB cable so that it's not tugging or potentially coming loose over time.

don't bunch up the cables too tightly as that leads to further trapped heat that will increase the overall running temp of the power cables. in your 24x7 environment, that heat may potentially have no 'room to breadth' and cool off or have the right thermal management scope.

This is awesome help, thanks very much @Ominence !!! Ok im going in for the fix:

- Replacing out all power cables for the stock EVGA (noticed my off-market white ones are around 20-22 AWG vs the 16 on the EVGAs - yikes)
- Ensure no tension on connections
- Swapping GPU 2 to PSU 2
- Am going to de-bundle cable runs & ensure good air circulation
- Im going to limit the power draw on the cards to something less intense.


Side note, i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous. Troubling because it means another issue but more hope that it could just be a shitty 24 pin.


Stupid question for you @Ominence, when powering up the PSUs should i switch on 1 or 2 first or try to do them simultaneous? Do not have them linked on power-up. I have been doing 1 then .5-1 second later #2.

 

[Ominence]

Those specific cards the OP is using can draw 420w peak gaming and when really pushed 450w ish.

that's were you check your tailored workload and ensure that the frequency doesn't exceed the manageable power draw, if you're having issues with the system, even if it has to be stock or below as the OP recognises. the 3090 is only rated for 1.395Ghz base and 1.695Ghz boost.
having said that, the pcie bus slot power will still not be excessive, even at 450w (single card case of course). you'll end up having 3x 8pins from the factory on those cards.
with a 4x multiplier buff, a bit of caution is obviously required.
GPUTweakII will easily manage any power draw issues those 3090s are having.

 

[FrankJames]

3090s can draw well over 350w though. Those specific cards the OP is using can draw 420w peak gaming and when really pushed 450w ish. And gaming loads are not constant so if you're slamming them with a constant workload... yikes.

Yes that's a good point, however with the type of models we are training they don't run the memory & cores simultaniouslly. Typically you max-out memory for a few mins loading the data-set up and once complete, it start processing in the cores.

I think I remember running them around 270-300w RMS when running hard.

that's were you check your tailored workload and ensure that the frequency doesn't exceed the manageable power draw, if you're having issues with the system, even if it has to be stock or below as the OP recognises. the 3090 is only rated for 1.395Ghz base and 1.695Ghz boost.
having said that, the pcie bus slot power will still not be excessive, even at 450w (single card case of course). you'll end up having 3x 8pins from the factory on those cards.
with a 4x multiplier buff, a bit of caution is obviously required.
GPUTweakII will easily manage any power draw issues those 3090s are having.

Any tips for good software for GPU tuning on Linux?

Also, do you have a recommended, rough idea of how I should reduce power load on tune? I was thinking:
- Reduce core clock down to 1.3
- Reduce voltage 10%
- Reduce boost down to 1.4-1.5
- Reduce power limit by 15%

 

[Ominence]

Stupid question for you @Ominence, when powering up the PSUs should i switch on 1 or 2 first or try to do them simultaneous? Do not have them linked on power-up. I have been doing 1 then .5-1 second later #2.

Connect 2 PSUs to 1 Motherboard? — Quick How-To Guide - YouTube

please ensure that the AWG of the bridging tool is correct too!

 

[GerKNG]

i saw your post yesterday and it was deleted a few minutes later.

your PSUs area basically a high wattage low end PSU (EVGA GX PSUs are absolute garbage..)

just get two AX1600I Titanium from Corsair or two bequiet! dark power pro 12 1500W. (since your System costs as much as a decent used car.)

 

[Ominence]

you can always use a U wire (glued in, and insulated on the exposed part) to bridge the stock 24pin on PSU2 and before you fire PSU1, fire up PSU2 1st after the bridging.

 

[thesmokingman]

Connect 2 PSUs to 1 Motherboard? — Quick How-To Guide - YouTube

please ensure that the AWG of the bridging tool is correct too!

View attachment 209016View attachment 209017

That's all I needed when I ran dual psu. If you look closely you can see the jumper cable.

 

[Ominence]

Any tips for good software for GPU tuning on Linux?

GreenWithEnvy—Linux Apps on Flathub

Also, do you have a recommended, rough idea of how I should reduce power load on tune? I was thinking:
- Reduce core clock down to 1.3
- Reduce voltage 10%
- Reduce boost down to 1.4-1.5
- Reduce power limit by 15%

reducing voltage will just make the system potentially less stable.

power is a function of V and I. between them, they can still get to the hypothetical 450w, stability permitting. you should try to work on the much simpler power limit settings if the Linux tool allows it to go negative from vendor stock.

Edit- if the PL option is not available, reduce the boost clock 1st (to stock) as that is the stretch target on the clock speed algorithm.

 

[looniam]

Side note, i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous. Troubling because it means another issue but more hope that it could just be a shitty 24 pin.

there *should be* 5 lines of 12v in the riser cable:

at 1.1 amps each for ~66 watts, the other ~9 is 3.3v (75 total)

that riser, esp w/o getting hot, is perfectly fine . . . it seems.

 

[Ominence]

Side note, i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous. Troubling because it means another issue but more hope that it could just be a shitty 24 pin.

very likely that has been the issue with your nice setup. going for aesthetics, you have to be very careful with all technical aspects too

just FYG, my FD flex is rated to 40w 3.33A.
Flex VRC-25 _Product sheet (fractal-design.com)
cable run length would be an issue for you though.

i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous.

not to be alarmist, but they may have already burned out without showing through the insulation. just to be sure, do a simple continuity check on any common multimeter. they look like passive cables without additional electronic circuitry running on them.

That's all I needed when I ran dual psu. If you look closely you can see the jumper cable.

ah, reminds me of the days i used to do quad CFX systems before going to solitary Titans.

 

[newtekie1]

Yep, sure did!

It's hard to tell what is going on in the pictures, but it isn't plugged in in the one picture that shows the connector. Are you sure it was plugged in?

 

[eidairaman1]

Yep, sure did!


Hey! Yes I have pics during build and after complete. LMK if you can see what you need


Good point, will try this as part of the fix


wait, took this video, this should cover everything (screenshots)


- Pump Draw: So the Pumps (while 12v) are 23W max load. We never max them out at full load so I would say around ~60W maximum. But I have an active ROG pump splitter in the back (seen in image 1830 at the bottom left) so I doubt its that?
- Fan Draw: Same; has an active splitter powering all fans.
- Load balancing: Interesting. So I have calculated that with all pumps, fans, MoBo, CPU pull around 300-400w when active. So with current set up im estimating load at max GPU usage to be:
PSU 1: 1100-1300 W
PSU 2: 700 W
If I load up 3 GPUs on PSU 2 I will be looking at:
PSU 1: 800-1100W
PSU 2: 750 W
Feel like its negligible difference?
- Overclocking: CPU has no OC, RAM is minor but will back
- Power: I'm pretty sure RAM and Threadripper does not use the 12v rail. From what I understand, the GPUs, Fans & Pumps all use the 12v rail but nothing else. Since only the 12v pins on the main MoBo cable were burned out, and all the fans and pumps are on Molex power from PSU 1 (with no issue) it leads me to think its a PCIe power bus overload (runs on 12v). I do have 2 SSds, one in DIMM 2 one in an M2.2 slot and im pretty sure DIMM 2 is pulling from the PCIe bus bout doubt its significant.


Yes but a combined PCIe bus draw is well above 100w. Did some more research and the 3090 OC can pull up to 400w at max-out. if the 2 8pin per GPU provide 300W combined power and If thats the case (even during transients) then im looking at a combined 400w pull from the PCIe bus JUST from the GPUs... Am i wrong?


That makes sense, will swap to all the stock cables. I'm pretty sure the AWG is a much higher rating on the stock. My only concern is regarless of this, I'm not sure I like the fact that i'm putting so much strain on the 12v rail regardless. The PSU can handle it fine but (I Think) but im concerned the MoBo & its PCIe bus is under serious strain.I Know the Zenith extreme Alpha is a beast but i'm looking for long-term reliability.

So:
- Swap out all the shitty power cables with stock
- Undervolt the cards?
- Perhaps I shift the active fan & pump splitters to PSU2 just to ensure a load reduction on PSU1 12v? If I think about it:
- PSU1 12v rail running: GPU 1 & 2, MoBo 12v (PCIe bus), active fan splitter, active pump splitter.
- PSU2 running: GPU 3 & 4.
- Only problem is im a litle concerned about having the MoBo (fan & pump headers) on one PSU and the splitters on another.....


Hey thanks for this! Do you mean on the PCIe power bus?

Yes for those cables to fry, it tells me the current draw is higher than what the cables can handle.